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ABSTRACT 


This  thesis  examines  the  use  of  random  sampling  as  a 
basis  for  taking  extrapolated  deductions  in  the  payment  of 
Navy  maintenance  service  contracts.  Motivated  by  increasing 
inspection  requirements,  NAVFAC  has  authorized  limited  use 
of  this  quality  assurance  tool.  The  experience  of  two 
activities  which  implemented  extrapolated  deductions  in 
their  family  housing  maintenance  contracts  is  the  focus  of 
this  study.  Methods  employed  are  evaluated  along  with  an 
analysis  of  payment  history,  contractor  performance  and 
inspection  requirements.  The  statistical  basis  used  in  the 
field  tests  along  with  an  alternate  proposal  is  analyzed. 
Various  issues  are  identified  which  should  be  addressed 
before  general  guidance  is  promulgated.  Major 
recommendations  deal  with  sample  adjustment,  computer 
support,  inspection  methods  and  the  need  for  oversight. 


3 


TABLE  OF  CONTENTS 


I.  INTRODUCTION  -  8 

A.  BACKGROUND -  8 

B.  OBJECTIVE  - - 9 

C.  SCOPE  AND  ASSUMPTIONS  -  9 

D.  RESEARCH  METHODOLOGY  -  9 

E.  THESIS  ORGANIZATION  -  10 

II.  MAINTENANCE  SERVICE  CONTRACTING  ENVIRONMENT  -  12 

A.  BACKGROUND -  12 

B.  TREND  TOWARDS  SERVICE  CONTRACTING  -  13 

C.  PERFORMANCE -  15 

D.  QUALITY  ASSURANCE  -  15 

E.  ENFORCING  PERFORMANCE  -  16 

III.  HISTORICAL  OVERVIEW  OF  STATISTICAL 

QUALITY  CONTROL  -  20 

A.  EARLY  EVOLUTION -  2  0 

B.  INFLUENCE  OF  WORLD  WAR  II -  2  2 

C.  MILITARY  STANDARDS  (MIL-STD-105 )  -  24 

1.  Development -  24 

2.  Acceptance  Sampling  -  25 

3.  Application  to  Air  Force  Service 

Contracting -  26 

4.  Navy  Rejection  of  MIL-STD-105D -  29 

IV.  RANDOM  SAMPLING  FOR  EXTRAPOLATED  DEDUCTIONS  -  31 

A.  INTRODUCTION -  31 


4 


B.  RANDOM  SAMPLING -  3  2 

1.  Characteristics -  32 

2.  Advantages -  3  3 

3.  Problems  in  Practical  Use -  33 

C.  EXTRAPOLATED  DEDUCTIONS  -  36 

1.  Background -  3  6 

2.  Legal  Considerations  -  37 

D.  STATISTICAL  BASIS  FOR  FIELD  TESTING  -  40 

1.  Introduction -  40 

2.  Use  of  the  Tables -  41 

3.  Mathematical  Basis  of  Field  Method  -  43 

4.  Effect  of  CPL -  44 

5.  Effect  of  Sample  Size -  45 

6.  Effect  of  Population  Estimate  -  47 

7.  Operating  Characteristics  -  48 

E.  PROPOSED  STATISTICAL  BASIS  FOR  FUTURE  USE  -  50 

F.  COMPARISON  OF  THE  TWO  STATISTICAL  BASES  -  53 

1.  Inspection  Requirements  -  53 

2.  Effect  on  Contractor  Performance  -  55 

3 .  Deduction  Level -  56 

4.  Producer/Consumer  Risk -  57 

5.  Ease  of  Use -  58 

V.  SURVEY  OF  FIELD  TESTS -  59 

A.  INTRODUCTION -  59 

1.  Purpose -  59 

2 .  Background -  59 

5 


3 .  Data  Collection -  60 

B.  CONTRACT  REQUIREMENTS  -  61 

1.  Treatment  of  Service  Calls  at  Great 

Lakes -  61 

2.  Treatment  of  Service  Calls  at  San 

Francisco -  62 

C.  QUALITY  ASSURANCE  PLANS  -  63 

1.  Introduction -  63 

2.  General  Comments  -  63 

3 .  Critical  Performance  Levels  -  64 

4.  Performance  Weighting  -  65 

D.  INSPECTION  REQUIREMENTS  -  67 

1.  Introduction -  67 

2.  Sampling  Levels -  67 

3 .  Surveillance  Level  Switching  -  69 

4.  Sample  Generation  -  71 

5.  Sample  Adjustment  -  71 

E.  PAYMENT  ANALYSIS  -  73 

1.  Introduction -  73 

2.  Payment  Methodology  -  73 

3.  Treatment  of  Liquidated  Damages  -  75 

4 .  Treatment  of  Customer  Complaints  -  77 

5.  Deduction  Data -  77 

6.  Effect  on  Performance -  79 

7.  Comparison  with  RSED  V3 . 0 -  81 

VI.  CONCLUSIONS  AND  RECOMMENDATIONS  -  83 

A.  GENERAL  ISSUES -  8  3 


6 


1.  Need  for  Oversight -  83 

2.  Treatment  of  Sample  Adjustment  -  84 

3.  Combining  Homogeneous  Populations  -  84 

4.  Surveillance  Levels  -  85 

5.  Performance/Payment  Match  -  86 

6.  Computer  Assistance  -  86 

B.  RSED  V3.0 -  87 

C.  RECOMMENDATIONS  FOR  FUTURE  STUDY  -  88 

APPENDIX  A:  USING  MIL-STD-105D  -  89 

APPENDIX  B:  MO-327  RANDOMS  SAMPLING  PROCEDURES  -  97 

APPENDIX  C:  SAMPLING  TABLES  USED  IN  THE  FIELD  TEST  -  108 

APPENDIX  D:  RSED  V3 . 0  TABLES  AND  PAYMENT  ANALYSIS  -  118 

LIST  OF  REFERENCES -  12  3 

INITIAL  DISTRIBUTION  LIST  -  125 


7 


I. 


INTRODUCTION 


A .  BACKGROUND 

Random  Sampling  for  Extrapolated  Deductions  (RSED) 
involves  making  an  inference  concerning  a  population  based 
on  a  simple  random  sample.  The  results  of  that  sample  are 
then  used  as  a  basis  for  equitable  payment  across  the  entire 
contracted  service.  As  a  simple  example,  suppose  that  a 
sample  inspection  of  services  revealed  10%  not  performed. 
The  contracting  officer  would  then  estimate  that  10%  of  all 
services  were  not  performed,  and  reduce  the  contract  price 
by  10%.  While  payment  deductions  for  observed  poor  perform¬ 
ance  or  nonperformance  by  service  contractors  is  standard 
practice,  the  extrapolation  of  deductions  based  on  random 
sampling  to  the  general  population  is  currently  prohibited 
by  NAVFAC  regulation.  This  policy  has  been  under  review  in 
recent  years. 

Interest  in  this  technique  is  a  result  of  the  increasing 
role  that  service  contracts  play  in  accomplishing  the  base 
maintenance  function  and  the  subsequent  strain  placed  on  the 
inspection  force.  As  part  of  a  move  toward  formulating  Navy 
policy,  two  activities  were  granted  authority  to  use 
extrapolated  deductions  in  their  Family  Housing  Maintenance 
Contracts. 
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B.  OBJECTIVE 


The  objective  of  this  research  is  to  evaluate  the  field 
use  of  random  sampling  for  extrapolated  deductions  in  Navy 
maintenance  service  contracts.  This  is  being  done  to 
identify  issues  which  should  be  addressed  before  general 
authority  is  granted  and  guidance  promulgated.  In  addition, 
proposed  NAVFAC  statistical  methodology  is  evaluated  in 
light  of  information  obtained  in  the  course  of  this  study. 

C.  SCOPE  AND  ASSUMPTIONS 

This  study  focuses  on  the  procedures  used  to  implement 
the  field  testing  of  random  sampling  for  extrapolated 
deductions  in  Navy  maintenance  service  contracts.  Air  Force 
use  of  this  contracting  technique  is  addressed  to  provide  a 
background  and  contrast  to  the  NAVFAC  effort.  Experiences 
of  other  service  contracting  organizations  with 
extrapolating  deductions  either  within  or  outside  the 
Federal  Government  was  not  pursued.  It  is  assumed  that  the 
reader  is  familiar  with  the  concept  of  simple  random 
sampling. 

D.  RESEARCH  METHODOLOGY 

A  literature  search  was  conducted  to  review  regulations, 
reports,  industrial  literature,  and  policy  guidance  applica¬ 
ble  to  statistical  quality  assurance  and  service 
contracting.  Contact  documentation  and  history  was  obtained 
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from  the  following  organizations  involved  in  the  initial  use 
of  extrapolating  deductions: 

1.  Public  Works  Center,  San  Francisco,  Ca;  and 

2.  Public  Works  Center,  Great  Lakes,  II. 

Additionally,  interviews  were  conducted  either  in  person 

or  by  telephone  at  the  following  organizations: 

1.  Public  Works  Center,  San  Francisco,  Ca; 

2.  Public  Works  Center,  Great  Lakes,  II; 

3.  Western  Division,  Naval  Facilities  Engineering 
Command,  San  Bruno,  Ca; 

4.  Public  Works  Center,  San  Diego,  Ca; 

5.  Public  Works  Department,  Naval  Postgraduate  School, 
Monterey,  Ca; 

6.  Public  Works  Department,  Marine  Corps  Air  Station,  El 
Toro,  Ca; 

7.  Vandenburg  Air  Force  Base,  Lompoc,  Ca. 

E.  THESIS  ORGANIZATION 

This  chapter  identified  the  current  status  of  NAVFAC 
regarding  the  extrapolation  of  deductions  in  service  con¬ 
tracting  and  presents  the  objectives  and  methodologies  of 
this  study.  Chapter  II  puts  the  study  into  perspective  with 
a  discussion  of  the  maintenance  service  contracting  environ¬ 
ment.  Chapter  III  traces  the  historical  development  of 
statistical  quality  assurance  techniques  and  their  applica¬ 
tion  to  service  contracting.  Chapter  IV  examines  random 
sampling,  NAVFAC  policy  concerning  RSED,  and  two  statistical 
approaches  that  have  received  consideration  as  a  basis  for 
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extrapolating  deductions  in  NAVFAC  service  contracts. 
Chapter  V  presents  findings  and  conclusions  resulting  from 
an  evaluation  of  two  activities  implementation  of 
extrapolated  deductions  in  their  family  housing  maintenance 
contracts.  Finally,  Chapter  VI  lists  recommendations  for 
the  general  use  of  extrapolating  deductions  and  makes 
suggestions  for  future  research. 
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II .  MAINTENANCE  SERVICE  CONTRACTING  ENVIRONMENT 


A.  BACKGROUND 

A  Maintenance  Service  Contract  (MSC) ,  commonly  referred 
to  as  simply  a  service  contract,  calls  for  a  contractor's 
time  and  effort  rather  than  a  finished  product.  Service 
contracts  are  funded  by  the  organization  having  budgeted 
responsibility  for  the  services  acquired.  Examples  of 
service  contracts  include  refuse  collection,  guard  services, 
grounds  maintenance,  pest  control,  and  family  housing 
maintenance.  A  MSC  is  normally  entered  into  for  a  one  year 
period  with  an  option  to  extend  for  two  additional  one  year 
periods.  However,  extensions  to  the  contract  cannot  exceed 
the  fifth  anniversary  of  contract  award.  [Ref.  1] 

While  not  a  critical  distinction  for  purposes  of  this 
paper,  a  MSC  is  one  category  of  a  Facility  Support  Contract. 
The  other  category  is  a  Maintenance  Construction  Contract. 
Due  to  the  nonrepetitive  nature  of  construction  and  the 
higher  degree  of  importance  associated  with  each  unit  of 
work,  they  are  not  normally  amenable  to  statistical  quality 
assurance  and  will  not  be  a  focus  of  this  study. 

Facility  Support  Contracts  are  designed  to  accomplish 
the  maintenance  and  repair  of  real  property  facilities. 
They  can  best  be  thought  of  as  an  extension  of  the  base 
public  works  force,  and  in  fact  replace  the  in-house  forces 
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where  a  decision  has  been  made  to  contract  out  a  function 


previously  performed  by  Public  Works. 

Service  contracts  are  almost  exclusively  of  the  fixed 
price  type.  It  is  the  belief  of  NAVFAC  that  fixed  price 
contracts  force  contractors  to  perform  efficiently.  Where 
uncertainty  of  workload  precludes  a  firm  fixed  price, 
Indefinite  Quantity  contracts  are  awarded  on  the  basis  of 
firm  unit  prices  for  an  estimated  workload.  This  study 
encountered  the  extrapolation  of  deductions  being  applied 
only  to  firm  fixed  price  contracts,  however  the  technique  is 
also  applicable  to  firm  unit  prices.  [Ref.  1] 


B.  TREND  TOWARDS  SERVICE  CONTRACTING 

Service  contracting  in  the  public  sector  gained 

attention  in  1954  when,  in  his  first  budget  message  to 

Congress,  President  Eisenhower  said:  [Ref.  2] 

This  budget  marks  the  beginning  of  a  movement  to  shift  to 
.  .  .  private  enterprise  Federal  activities  which  can  be 
more  efficiently  carried  on  that  way. 

This  was  followed  by  Bureau  of  the  Budget  Bulletin  55-4 

which  implemented  the  stated  policy  as  follows:  [Ref.  3] 

It  is  the  general  policy  of  the  administration  that  the 
Federal  Government  will  not  start  or  carry  on  any  commer¬ 
cial  activity  to  provide  a  service  or  product  for  its  own 
use  if  such  service  or  product  can  be  produced  from 
private  enterprise  through  ordinary  business  channels. 
Exceptions  to  this  policy  shall  be  made  by  the  head  of  the 
agency  only  where  it  is  clearly  demonstrated  in  each  case 
that  it  is  not  in  the  public  interest  to  procure  such 
products  or  services  from  private  enterprise. 

Today's  policy  comes  from  OMB  Circular  A-76,  issued  in 
1966  and  further  revised  in  1967,  1979  and  1983.  It 
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basically  states  that  whenever  a  commercial  activity  is 
Federally  operated,  the  cost  of  operation  should  be  compared 
to  the  cost  of  contracting  to  determine  who  should  do  the 
work.  A  commercial  activity  (CA)  is  defined  as  a  product  or 
service  obtainable  from  a  commercial  source.  [Ref.  4] 

Although  it  has  been  federal  policy  to  contract  out 
commercial  functions  since  the  1950s,  the  program  was 
largely  ignored  until  the  1979  revision  of  A-76  [Ref.  5].  A 
key  element  of  that  version  required  government  forces  to 
submit  a  "firm  bid"  based  on  a  clear  Statement  of  Work 
(SOW) .  Contractors  would  also  bid  on  the  requirements 
contained  in  the  SOW.  Thus  the  open  market  would  determine 
the  contracting  decision.  Award  would  be  to  the  lowest 
responsible  bidder  if  under  the  government  estimate  by  10%. 
To  ensure  that  the  government  estimate  was  properly 
prepared,  it  was  subject  to  approval  by  the  Naval  Audit 
Service  [Ref.  6]  . 

Thus,  the  fact  that  DODI  4100.3  3  required  a  complete 
review  of  all  CA  functions  and  a  compliance  system  was  in 
place  set  the  stage  for  a  number  of  functions  formally 
performed  by  Public  Works  to  be  contracted  out. 

Data  compiled  in  January  1982  showed  that  60%  of  all 
functions  reviewed  shifted  to  contract,  at  an  overall  cost 
savings  of  19%.  Of  all  NAVFAC  functions  studied,  the  areas 
of  transportation,  grounds  maintenance,  building  mainte¬ 
nance,  and  janitorial  service  accounted  for  82%  of  the  total 
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[Ref.  7].  The  next  few  years  will  undoubtedly  see  conver¬ 
sions  occur  at  an  ever  increasing  rate  as  the  great  majority 
of  reviews  are  completed. 

C.  PERFORMANCE 

Service  contracts,  while  not  directly  mission  related, 
are  usually  of  great  concern  to  the  activity  commanding 
officer.  Visible  lack  of  performance  reflects  directly  on 
his  ability  to  effectively  maintain  the  shore  establishment 
entrusted  to  his  care.  The  result  is  acute  command  interest 
in  performance,  an  interest  which  is  reflected  throughout 
the  contracting  organization. 

Acceptable  performance  in  service  contracts  is  often 
difficult  to  achieve.  While  a  consensus  of  opinion  did  not 
emerge  during  personal  interviews,  frequently  cited  reasons 
included  a  lack  of  inspector  resources,  inadequate  specifi¬ 
cations,  and  a  lack  of  enforcement  techniques  to  motivate 
the  contractor. 

D.  QUALITY  ASSURANCE 

Quality  Assurance  (QA)  is  the  procedure  by  which  the 
government  verifies  that  it  is  receiving  the  services  for 
which  it  contracted  and  paid  for.  Quality  Assurance 
Evaluators  (QAE)  are  responsible  to  the  Service  Contract 
Manager  (SCM)  for  actual  monitoring  of  contractor 
performance.  The  QA  program  is  contained  in  the  QA  Plan, 
it  details  how  the  QAE  will  monitor  all  aspects  of  a 
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contractor's  performance,  although  it  is  not  part  of  the 
actual  contract. 

Several  surveillance  methods  may  be  used  as  part  of  the 
QA  Plan.  Those  selected  are  based  on  the  unique  considera¬ 
tions  of  an  activity.  Techniques  currently  in  use  are: 
[Ref.  8] 

a)  One  Hundred  Percent  Inspection — Appropriate  for  infre¬ 
quently  performed  services  or  those  of  great  impor¬ 
tance,  it  is  expensive  and  time  consuming. 

b)  Planned  Sampling — Designed  to  inspect  some  part  of  the 
contract  requirement.  It  cannot  be  used  to  make  an 
overall  judgment  concerning  contract  performance,  but 
is  useful  where  performance  in  a  particular  area  or 
location  is  of  interest. 

c)  Unscheduled  Inspection — An  impromptu  check. 

d)  Validated  Complaints — Originate  with  the  customer. 
The  method  is  successful  to  the  extent  that  customers 
are  aware  of  and  can  distinguish  good  performance. 
While  not  recommended  as  a  stand  alone  procedure,  it 
can  be  used  to  make  payment  deductions  and  require 
rework. 

e)  Random  Sampling — A  statistically  based  procedure  which 
attempts  to  incorporate  the  best  features  of  100% 
inspection  and  planned  sampling.  By  inspecting  a 
sample  representative  of  the  entire  population,  it 
allows  a  judgment  to  be  made  about  overall  performance 
with  a  reduced  level  of  inspector  resources.  A 
properly  constructed  random  sample  may  also  be  the 
basis  for  extrapolating  deductions.  However,  the 
extrapolation  of  deductions  is  currently  not  allowed 
by  NAVFAC  regulation  and  is  the  focus  of  this  study. 


E.  ENFORCING  PERFORMANCE 

If  the  government  accepts  anything  less  than  100%  of 
contract  bid  requirements,  the  contracting  officer  must  take 
some  appropriate  action.  Not  to  do  so  could  indicate  that 
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the  government  has  overstated  its  requirements  resulting  in 
an  illegal  contract. 

Other  than  the  routine  identification  of  contractor 
deficiencies,  negative  motivators  form  the  basis  of  NAVFAC 
approved  techniques  to  enforce  performance.  Contract 
actions  other  than  payment  reduction  are  all  designed  to 
threaten  and  support  eventual  default  for  failure  to 
perform.  The  effectiveness  of  these  techniques  are 
questionable  because  of  the  difficulty  in  sustaining 
termination  of  a  borderline  contractor.  In  addition,  the 
prospect  of  extended  litigation  and  interruption  in  services 
must  be  considered.  Conversations  with  contracting  officers 
indicate  that  "living"  with  less  than  acceptable  performance 
until  contract  expiration  is  the  best  of  a  bad  situation. 
It  is  reasonable  to  assume  that  a  savvy  contractor  is  aware 
of  the  contracting  officer's  dilemma. 

The  most  effective  performance  motivator  currently 
permitted  is  the  withholding  of  payment  for  unsatisfactorily 
performed  services.  Withholding  payment  assumes  a  strong 
profit  motive  on  the  part  of  a  capable  contractor,  a  "fair" 
contract  under  which  reasonable  profits  are  possible,  and  a 
schedule  of  deductions  which  results  in  reduced  profits  when 
exercised.  Other  possible  contractor  motivations  such  as 
growth  or  maintenance  of  the  workforce  do  not  appear  valid 
in  service  contracting.  Contractors  generally  do  not  staff 
for  a  job  until  contract  award,  and  a  ready  supply  of 
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unskilled  or  semiskilled  laborers  makes  adjustment  of  the 
workforce  an  easy  thing  to  do. 

Current  NAVFAC  regulations  allow  deductions  for  work  not 
acceptably  performed,  unless  the  contractor  is  given  the 
opportunity  to  re-perform  the  work  and  does  so  satisfactor¬ 
ily.  If  feasible,  rework  is  preferable  to  deductions 
because  of  the  emphasis  on  performance.  When  taken,  deduc¬ 
tions  are  in  accordance  with  individual  bid  items  if  an 
Indefinite  Quantity  contract,  or  an  agreed-upon  schedule  of 
deductions  for  Firm  Fixed  Price  contracts.  However,  this 
contract  action  is  limited  by  the  fact  that  appeals  boards 
and  courts  uniformly  hold  "penalty"  provisions  unenforce¬ 
able.  Section  9-302.1  of  the  Navy  Contracting  Manual 
applies:  [Ref.  1] 

No  contract  may  provide  for  penalties  for  non-performance, 
as  such  are  legally  unenforceable.  However,  a  contract 
may  provide  for  liquidated  damages,  which  is  an  advance 
agreement  between  parties  to  a  contract  as  the  damages  one 
party  will  suffer  if  the  other  fails  to  perform  .  .  .  Note 
that  if  a  contractor  does  not  perform  work,  it  would  be 
improper  to  withhold  payment  for  that  work  AND  also  charge 
him  the  full  cost  of  having  the  work  done  by  others,  for 
that  would  result  in  the  Navy  having  work  done  at  the 
expense  of  others. 

Liquidated  damages  of  10%  are  assessed  whether  or  not 
the  work  is  subsequently  performed  by  the  contractor  to 
compensate  the  government  for  increased  administrative 
costs.  Where  government  forces  re-perform  the  work,  a  20% 
liquidated  damage  charge  is  levied  along  with  the  full  cost 
of  the  government  effort.  [Ref.  1] 
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Current  NAVFAC  regulations  allow  deductions  for  observed 
defects  only.  The  extrapolation  of  deductions  based  on 
random  sampling  is  prohibited  [Ref.  8].  This  policy 
resulted  from  the  lack  of  an  acceptable  statistical  tech¬ 
nique  and  questions  as  to  the  legality  of  extrapolated 
deductions.  However,  NAVFAC  has  recognized  the  potential  of 
extrapolating  deductions  to  further  motivate  the  contractor, 
or  at  least  to  better  match  payment  for  services  received. 
Therefore  they  have  initiated  efforts  in  recent  years  to 
define  a  methodology  under  which  extrapolation  might  be 
authorized  for  general  use.  In  support  of  this  effort, 
NAVFAC  has  authorized  trial  tests  of  extrapolating  deduc¬ 
tions  in  family  housing  maintenance  contracts  at  PWC  San 
Francisco  and  PWC  Great  Lakes. 
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III.  HISTORICAL  OVERVIEW  OF  STATISTICAL 
QUALITY  CONTROL 


A.  EARLY  EVOLUTION 

Since  man  began  to  manufacture  and  produce  products, 
there  has  been  concern  over  quality  of  output.  As  far  back 
as  the  Middle  Ages,  the  medieval  guilds  insisted  on 
apprenticeships  and  required  that  those  seeking  to  become 
master  craftsmen  offer  evidence  of  their  ability.  Such 
rules  were  in  part  a  form  of  quality  control.  In  more 
modern  times,  factory  inspection  and  research,  activities  of 
professional  societies,  and  various  regulations  have  sought 
for  years  to  assure  a  quality  of  output.  Quality  control 
has  thus  had  a  long  history. 

On  the  other  hand,  statistical  quality  control  is  rela¬ 
tively  new.  The  science  of  statistics  itself  goes  back  only 
about  300  years.  The  theory  of  probability  originated  in 
1654  when  Pascal,  a  French  philosopher  and  mathematician, 
teamed  up  with  Pierre  Fermat  to  develop  the  new  science. 
Early  applications  were  made  in  astronomy  and  physics,  but 
it  was  not  until  the  1920's  that  statistical  theory  began  to 
be  applied  effectively  to  quality  control.  It  should  be 
remembered  though,  that  even  before  the  1920's,  industry  was 
learning  to  do  things  more  scientifically.  Techniques  such 
as  the  GANTT  chart,  under  principles  developed  by  Taylor, 
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Gilbreth  and  others  were  a  part  of  the  movement  called 
"scientific  management." 

Walter  Shewhart,  of  the  Bell  Telephone  Laboratories,  was 
among  the  first  to  apply  statistical  methods  to  the  problem 
of  quality  control  [Ref.  1].  Two  other  Bell  System  men, 
H.F.  Dodge  and  H.G.  Romig,  took  the  leadership  in  developing 
the  application  of  statistical  theory  to  sampling 
inspection.  The  culmination  of  their  work  being  the  now 
well-known  Dodge-Romig  Sampling  Inspection  Tables.  An 
immense  amount  of  work  went  into  the  preparation  of  these 
tables,  for  at  that  time  hand  calculators  were  the  only 
available  aid  to  speedy  calculations.  These  tables  were 
published  in  1941  [Ref.  2].  The  work  of  Shewart,  Dodge,  and 
Romig  constitutes  much  of  what  today  comprises  the  theory  of 
statistical  quality  control. 

When  this  new  inspection  plan  was  proposed  there  were 
many  skeptics.  Therefore,  in  the  initial  trials,  after 
performing  sampling  inspection,  every  lot  was  then  inspected 
100%  to  check  conformity.  The  checks  showed  that  sampling 
did  give  good  results  and  with  much  less  time  and  money  used 
for  inspection.  After  its  initial  application  in  the 
central  offices  of  Western  Electric  Company,  the  new  proce¬ 
dures  were  tried  in  the  Hawthorne  manufacturing  plant  in 
Chicago,  and  proved  a  big  success. 

The  rate  of  adoption  of  these  newer  methods  were  slow  in 
the  United  States.  Professor  Freeman,  who  was  promoting 
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statistical  quality  control  at  Massachusetts  Institute  of 
Technology,  ascribed  this  sluggish  response  of  the  early 
years  to:  (1)  the  conviction  of  American  production 
engineers  that  their  principal  function  was  to  improve 
quality  control  through  advances  in  technology,  (2)  a 
prevailing  attitude  that  the  laws  of  chance  had  no  place  in 
a  scientific  field,  and  (3)  the  scarcity  of  adequately 
trained  industrial  statisticians.  In  1937,  probably  not 
more  than  a  dozen  single  enterprises  in  American  mass- 
production  industry  had  introduced  the  new  techniques  into 
their  ordinary  operations.  [Ref.  3] 

B.  INFLUENCE  OF  WORLD  WAR  II 

World  War  II  overcame  the  initial  coolness  of  American 
industry  toward  statistical  quality  control.  The  armed 
services  began  to  enter  the  market  as  large  consumers  of 
American  output  and,  as  such,  had  an  increasing  influence  on 
quality  standards.  A  group  of  engineers  from  Bell  Telephone 
were  brought  to  Washington  to  develop  a  sampling  inspection 
program  for  Army  Ordnance.  This  step  was  made  necessary  by 
the  bottlenecks  occurring  at  inspection  stations  during  full 
scale  production. 

The  group  from  Bell  faced  a  sampling  problem  of  unprece¬ 
dented  size.  It  was  necessary  that  all  vendors  be  treated 
fairly;  it  was  necessary  to  get  enormous  quantities  of  ord¬ 
nance  material  quickly;  and  it  was  necessary  that  arms  and 
ammunition  be  reliable  and  safe.  The  "acceptable  quality 
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control"  system  that  was  developed  encompassed  the  concept 
of  protecting  the  consumer  from  getting  unacceptably  defec¬ 
tive  material,  and  encouraging  the  producer  in  the  use  of 
process  quality  control.  This  was  done  by  varying  the 
quantity  and  severity  of  acceptance  inspections  in  direct 
relation  to  the  importance  of  the  characteristics  inspected, 
and  in  inverse  relation  to  the  quality  level  indicated  by 
those  inspections  [REf.  4].  The  same  men  also  carried  out 
an  extensive  training  program  to  acquaint  government  person¬ 
nel  with  the  new  procedures  and  tables.  Many  terms  used 
extensively  today,  most  notably  "producers  risk"  and 
"consumers  risk,"  had  their  origins  in  this  original  effort. 

Wartime  research  in  statistical  quality  control  was 
carried  out  primarily  by  the  Statistical  Research  Group, 
Columbia  University,  Applied  Mathematics  Panel,  Office  of 
Scientific  Research  and  Development.  This  research  group 
was  organized  in  July  1942,  and  continued  in  existence  until 
September  1945.  Members  of  the  staff  were  drawn  from 
universities  around  the  country.  It  advised  and  assisted 
the  Army  and  Navy  on  statistical  aspects  of  problems  arising 
in  their  activities.  Included  in  its  contributions  to  sta¬ 
tistical  quality  control  was  the  preparation  of  the  Navy 
manual  on  sampling  inspection  by  attributes  [REf.  5]. 
Inspection  by  attributes  requires  that  the  inspected  item  be 
categorized  either  as  totally  acceptable  or  not  acceptable. 
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Professor  A.  Wald,  also  of  the  Statistical  Research 
Group,  developed  the  idea  of  sequential  sampling.  In 
sequential  sampling,  the  sample  size  is  not  determined  in 
advance.  The  sequential  procedure  is  to  inspect  one  item  at 
a  time.  The  inspection  is  continued  until  enough  cumulative 
evidence  has  been  gathered  to  either  accept  or  reject  a  lot. 
A  pre-assigned  risk  of  making  incorrect  decisions  controls 
the  ultimate  same  sample  size.  This  procedure  allows  very 
good  lots  to  be  accepted  quickly  and  bad  lots  to  be  rejected 
quickly.  It  is  only  with  lots  of  doubtful  quality  that 
extensive  sampling  is  required.  This  analysis  was  deemed  so 
important  by  the  U.S.  Government  that  it  withheld  publica¬ 
tion  of  Wald's  original  paper  until  June  1945.  [Ref.  5] 

C.  MILITARY  STANDARDS  (MIL-STD-105 ) 

1 .  Development 

In  the  post  war  period  there  were  two  systems  of 
sampling  plans  and  tables.  Army  tables  and  Navy  tables.  The 
procedures  and  tables  were  alike  in  general  plan  and  struc¬ 
ture  but  differed  in  a  number  of  respects.  In  1949,  a  basic 
Military  Standard,  MIL-STD-105,  was  developed  using  material 
from  both  service  tables.  Military  Standard  105D,  the 
current  version,  is  the  outcome  of  a  study  by  an  American- 
British-Canadian  working  group  that  sought  to  derive  a 
common  standard  for  the  three  countries.  MIL-STD-105D  was 
issued  by  the  U.S.  Government  in  1963.  In  1971,  it  was 
adopted  by  the  American  National  Standards  Institute  as  ANSI 
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Z1.4  and  in  1973,  it  was  adopted  by  the  International 
Organization  for  Standardization  as  ISO/DIS-2859 .  [Ref.  6] 

2 .  Acceptance  Sampling 

Acceptance  sampling  is  the  method  of  evaluating  a 
group  of  units,  drawn  from  a  production  lot,  in  order  to 
determine  the  acceptability  of  the  whole  lot.  It  has  been 
widely  accepted  as  the  preferred  method  of  acceptance  con¬ 
trol.  MIL-STD-105D  provides  tables  which  facilitate  the 
implementation  of  an  Acceptance  Sampling  by  Attributes  plan. 
The  focus  of  the  plan  is  an  acceptable  quality  level  (AQL) . 
An  AQL  represents  that  quality  level  which,  for  purposes  of 
sampling  inspection,  can  be  considered  satisfactory  as  a 
process  average.  [Ref.  15] 

In  addition  to  an  initial  decision  on  an  AQL,  it  is 
also  necessary  in  applying  MIL-STD-105D  to  decide  on  an 
"inspection  level."  This  determines  the  relationship 
between  the  lot  size  and  the  required  sample  size.  Three 
inspection  levels  are  provided  for  general  use  with  guidance 
for  switching  between  them. 

For  a  particular  sample  size  and  AQL,  the  tables 
will  specify  a  reject  number.  When  the  number  of  defects  in 
a  sample  is  equal  to  or  greater  than  the  reject  number,  the 
entire  lot  is  judged  to  be  unacceptable.  The  sample  size, 
reject  number,  and  AQL  define  an  "operating  characteristic" 
curve  for  a  particular  plan.  This  curve  depicts  the 
relationship  between  the  actual  defect  rate  submitted  for 
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inspection  and  the  corresponding  probability  of  acceptance 
of  such  lots  by  the  sampling  plan.  It  is  based  on  the 
binomial  distribution. 

The  nature  of  acceptance  sampling  is  explained  by 
Siegmund  Halpern:  [Ref.  15] 

Inasmuch  as  only  a  portion  of  the  whole  lot  is  inspected, 
there  is  always  a  risk  that  the  quality  of  the  sample  will 
not  reflect  the  quality  of  the  lot.  As  a  result,  produc¬ 
tion  lots  may,  at  times,  undeservably  be  rejected  and 
faulty  lots  accepted.  The  former  will  be  of  interest  to 
the  producer.  He  wants  to  minimize  the  risk  of  having 
good  lots  rejected.  This  risk  is  identified  as  the 
producer's  risk.  The  risk  of  accepting  bad  lots  in  a 
sampling  scheme  is,  of  course,  the  primary  concern  of  the 
consumer.  This  risk  is  called  consumer's  risk.  The 
method  by  which  both  the  producer's  risk  and  the 
consumer's  risk  are  quantified  statistically  is  by 
constructing  an  operating  characteristic  curve. 

Acheson  Duncan  also  comments:  [Ref.  9] 

It  is  to  be  emphasized  that  the  purpose  of  acceptance 
sampling  is  to  determine  a  course  of  action,  not  to 
estimate  lot  quality.  Acceptance  sampling  specifies 
procedures  that,  if  applied  to  a  series  of  lots,  will  give 
a  specified  risk  of  accepting  lots  of  a  given  quality. 

Appendix  A  illustrates  how  the  sampling  plans 
contained  in  MIL-STD-105D  are  used.  A  typical  operating 
characteristic  curve  is  also  presented. 

3 .  Application  to  Air  Force  Service  Contracting 

Random  sampling  as  a  quality  assurance  tool  for 
monitoring  contractor  services  was  introduced  to  the 
military  by  the  Air  Force  in  1979  with  AFR  400-28.  MIL-STD- 
105D  provided  the  basis  for  the  random  sampling  procedures 
contained  in  the  regulation.  [Ref.  16] 


26 


Originally  designed  to  determine  the  acceptance  or 
rejection  of  manufactured  lots,  MIL-STD-105D  was  easily 
adapted  to  the  service  contracting  environment.  Specific 
instances  of  work  could  be  judged  "defective"  or  acceptable, 
as  required  in  attribute  sampling,  the  same  as  a  manufac¬ 
tured  part.  Lot  size  was  normally  determined  by  the  number 
of  services  provided  in  a  month,  which  corresponded  to  a  pay 
period.  A  major  difference  between  the  use  of  acceptance 
sampling  for  manufactured  items  and  evaluating  service  con¬ 
tract  performance  is  that  a  "lot"  of  services  could  not  be 
totally  rejected  if  found  unacceptable,  since  it  had  already 
occurred.  Therefore,  contract  actions  based  on  the  results 
of  acceptance  sampling  must  recognize  that  some  satisfactory 
work  has  taken  place  for  which  the  contractor  is  due 
payment. 

AFR  400-28  contains  tables  derived  from  MIL-STD-105D 
that  show  acceptance  and  rejection  numbers  corresponding  to 
specified  acceptable  quality  levels  and  sample  sizes.  The 
acceptance  number  is  the  maximum  number  of  defects  allowed 
before  a  sample  is  considered  unsatisfactory.  When  the 
number  of  defects  in  the  sample  exceeds  the  acceptance 
number,  contractor  performance  is  deemed  unsatisfactory  and 
contract  payments  can  be  reduced  for  the  service  provided 
during  the  sample  period. 

Allowable  AQLs  are  limited  to  those  which  appear  in 
MIL-STD-105D .  AFR  400-28  calls  for  the  establishment  of 
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reasonable  AQLs  as  part  of  the  original  CA  solicitation. 
Thus  contractors  bid  on  performance  requirements  of  less 
than  100%  for  various  services.  Contractors  are  also 
provided  with  the  activities  QA  plan  as  an  enclosure  to  CA 
solicitations.  This  is  done  to  inform  prospective  bidders 
as  to  how  the  contract  will  be  monitored  and  to  help  them 
set  up  their  own  quality  control  program.  It  is  not, 
however,  a  part  of  the  contract. 

The  extrapolation  of  deductions  is  a  key  feature  of 
AFR  400-28.  When  the  random  sampling  method  of  surveillance 
is  used  and  the  service  is  rejected,  contractor  payments  may 
be  reduced  in  the  same  proportion  for  the  entire  lot  as  the 
percentage  of  defects  found  in  the  sample.  For  example,  if 
a  rejected  sample  disclosed  that  15%  of  an  inspected  service 
was  defective,  contract  payments  for  the  service  may  be 
reduced  by  15%.  However  if  the  number  of  defects  found  do 
not  equal  or  exceed  the  reject  number,  the  services  are 
judged  satisfactory  and  no  deductions  are  taken. 

Numerous  problems  have  been  experienced  at  the 
activity  level  in  implementing  AFR  400-28.  An  audit  con¬ 
ducted  by  the  Air  Force  Audit  Agency  in  1985  to  evaluate 
quality  assurance  of  base-level  service  contracts  reported 
the  following  findings:  [Ref.  17] 

a)  for  seven  of  twenty  contracts  reviewed,  QAEs  made 
errors  in  the  use  of  random  number  tables; 

b)  for  fifteen  of  twenty  contracts  reviewed.  QAEs  did 
not  correctly  use  random  sampling  techniques  due  to 
the  complexity  of  AFR  400-28; 
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c)  for  ten  of  twenty  contracts  reviewed.  QAEs  did  not 
properly  apply  accept/reject  numbers  due  to  the  com¬ 
plexity  of  the  AFR  400-28  acceptance  tables;  and 

d)  sampling  plans  were  not  an  effective  means  of  deter¬ 
mining  contractor's  performance  as  the  plans  used  fre¬ 
quently  required  the  acceptance  of  services  even 
though  the  percentage  of  defects  occurring  in  the 
sample  was  approximately  twice  the  defect  rate  per¬ 
mitted  in  the  contract. 

The  Air  Force  is  now  investigating  a  simplified 
random  sampling  process  based  on  Indifference  Quality 
Levels.  In  such  a  plan,  the  government  and  the  contractor 
equally  share  in  the  probability  of  acceptance  or  rejection 
at  a  specified  quality  level  [Ref.  17]. 

4 .  Navy  Rejection  of  MIL-STD-105D 

NAVFAC  service  contracts  are  written  requiring  100% 
performance.  Thus  the  Navy  is  more  interested  in  a  statis¬ 
tical  methodology  which  will  provide  information  as  to 
actual  contract  performance  rather  than  the  accept/reject 
hypothesis  testing  of  MIL-STD-105D. 

A  random  sampling  procedure  was  designed  by  NAVFAC 
in  1979  to  determine  the  level  of  contractor  performance. 
Published  in  the  MO-327  [Ref.  8],  which  is  the  Navy  equiva¬ 
lent  of  AFR  400-28,  it  was  not  intended  to  be  a  statistical 
basis  for  extrapolating  deductions.  The  sampling  plan 
contained  in  MO-327  made  use  of  an  AQL  that  was  not  releasa¬ 
ble  to  the  contractor.  This  AQL  differed  from  the  one  used 
in  MIL-STD-105D  in  that  it  was  defined  to  be  "tolerable" 
performance  rather  than  acceptable. 
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The  new  contract  surveillance  technique  was  not 
widely  used.  Conversations  with  contract  administrators 
indicated  that  this  method  of  random  sampling  was  hard  to 
understand,  difficult  to  administer,  and  provided  question¬ 
able  results  upon  which  to  base  contract  actions.  Adding  to 
that  the  fact  that  extrapolated  deductions  were  not 
permitted,  there  remained  little  incentive  for  its  use  over 
the  more  understandable  planned-sampling  technique. 

In  1982,  NAVFAC  determined  that  the  extrapolation  of 
deductions  was  appropriate  as  long  as  there  was  a  degree  of 
confidence  that  deductions  made  were  reasonable  and  not 
construed  as  penalties.  Their  approach  is  studied  in  the 
following  chapter. 
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IV.  RANDOM  SAMPLING  FOR  EXTRAPOLATED  DEDUCTIONS 


A.  INTRODUCTION 

Random  Sampling  for  Extrapolated  Deductions  (RSED) 
involves  making  an  inference  concerning  a  population  based 
on  a  simple  random  sample.  The  results  of  that  sample  are 
then  used  as  a  basis  for  equitable  payment  across  the  entire 
contracted  service. 

There  are  three  reasons  why  the  concept  of  extrapolating 
deductions  is  supported  by  NAVFAC  at  this  time:  (1)  it  is 
believed  to  provide  extra  motivation  for  the  contractor 
through  the  potential  of  increased  deductions,  (2)  extrapo¬ 
lation  results  in  a  better  match  of  payment  to  services 
received,  and  (3)  the  ability  to  extrapolate  deductions 
makes  the  use  of  random  sampling  more  attractive  as  a 
quality  assurance  tool.  Given  the  current  service 
contracting  environment,  the  ability  to  obtain  acceptable 
evaluation  of  contractor  performance  with  reduced  levels  of 
inspector  resources  would  appear  to  be  the  predominant 
issue . 

The  application  of  extrapolated  deductions  must  be  based 
on  a  statistically  valid  random  sample  if  a  protest  is  to  be 
upheld.  Most  activities  have  had  no  experience  with  random 
sampling  and  the  guidance  available  is  limited  in  value. 
The  next  section  looks  at  the  considerations  involved  in 
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employing  random  sampling  in  a  service  contracting 
environment . 

B.  RANDOM  SAMPLING 

1.  Characteristics 

A  properly  constructed  random  sample  must  ensure 
that  every  possible  sample  of  a  given  size  has  an  equal 
chance  of  being  selected.  Additionally,  the  process  being 
inspected  must  be  homogeneous  and  produce  defects  in  a 
random  manner.  If  this  is  not  the  case,  a  selection  bias 
may  result  which  causes  the  attributes  of  the  sample  to  be 
different  from  the  population.  [Ref.  18] 

This  theoretical  ideal,  approachable  in  manufactured 
lots,  cannot  realistically  be  attained  in  service 
contracting.  Some  rooms  are  going  to  be  harder  to  clean 
than  others.  A  bus  service  will  be  more  likely  to  lag 
behind  schedule  during  certain  hours  of  the  day.  If  grounds 
maintenance  is  provided  at  two  different  bases,  a  SCM  would 
not  be  happy  with  a  sample  which  required  95%  of  the 
inspections  at  one  base.  Thus,  a  sampling  plan  must  be 
devised  which  will  "reasonably"  represent  the  population  of 
services.  A  general  rule  would  be  to  adopt  a  selection 
method  that  will  give  every  member  of  the  population  an 
equal  chance  of  being  selected.  In  general,  it  is  best  to 
use  random  numbers.  These  will  hold  up  in  court  [Ref.  9]. 
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2 .  Advantages 


The  use  of  random  sampling  as  a  quality  assurance 
tool  has  inherent  advantages  over  other  inspection  tech¬ 
niques.  The  most  obvious  is  that  it  provides  a  cost  effec¬ 
tive  way  of  estimating  the  true  level  of  contractor 
performance.  The  fact  that  it  may  now  be  used  as  a  basis 
for  extrapolating  deductions  makes  it  all  the  more  attrac¬ 
tive.  Some  other  benefits  of  random  sampling  offered  by 
contract  administrators  are: 

a)  Requires  less  supervision  of  inspectors  to  make  sure 
they  are  using  their  time  effectively; 

b)  is  not  susceptible  to  the  predictability  which  may  be 
involved  in  planned  sampling; 

c)  allows  inspection  of  services  to  be  transferred  easily 
from  one  inspector  to  another  as  no  previous  knowledge 
of  the  contract  is  required. 

NAVFAC  has  recommended  that  random  sampling  be 
applied  to  service  contracts  possessing  the  following 
characteristics:  [Ref.  8] 

a)  Composed  of  a  large  homogeneous  population  of  work 
units , 

b)  individual  work  items  not  of  a  critical  nature, 

c)  population  size  estimable, 

d)  inspector  travel  time  not  excessive. 

3 .  Problems  in  Practical  Use 

Although  there  would  seem  to  be  numerous  advantages 
involved  with  the  use  of  random  sampling,  even  without 
extrapolated  deductions,  its  use  seems  to  be  very  limited 
within  the  NAVFAC  community.  Conversations  with  various 
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contract  administrators  indicate  that  a  lack  of  familiarity 
with  random  sampling  procedures,  the  complexity  of  writing 
amenable  specifications,  and  the  confusing  guidance  in  the 
MO-327  are  at  the  heart  of  their  reluctance. 

Probably  the  major  impediment  to  instituting  random 
sampling  plans,  with  or  without  extrapolated  deductions,  is 
the  lack  of  random  sampling  guide  specifications  for 
individual  contract  types.  Contract  administrators  are 
either  not  willing  or  able  to  get  bogged  down  in  the  details 
of  generating  performance  specifications  and  quality 
assurance  plans  which  properly  incorporate  random  sampling. 
For  that  reason,  a  great  many  services  amenable  to  random 
sampling  are  being  inspected  in  a  judgmental  manner  with 
poor  performance  defined  as  an  intolerable  level  of  customer 
complaints . 

Janitorial  services  would  seem  to  be  a  prime  candi¬ 
date  for  random  sampling,  yet  interviews  with  contract 
administrators  revealed  confusion  and  frustration  where 
attempts  to  write  a  suitable  plan  were  made.  This  confusion 
is  easily  understood.  Questions  exist  over  what  the  sampled 
unit  should  be.  Should  it  be  individual  rooms,  groups  of 
rooms,  buildings,  or  the  specific  acts  of  emptying  ashtrays 
and  cleaning  windows?  Also,  how  is  a  representative  sample 
taken  when  conditions  vary?  Suppose  that  a  BOQ  building  is 
randomly  selected  for  inspection.  Can  the  percent  of  rooms 
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found  to  be  nonconforming  be  used  as  an  indicator  of  overall 
performance  across  the  various  buildings  on  base? 

The  design  of  random  sampling  plans  is  beyond  the 
scope  of  this  paper.  It  is  addressed  because  of  its 
discovery  as  an  issue  in  the  course  of  the  research.  Due  to 
the  complexities  involved  with  designing  random  sampling 
plans  and  the  lack  of  specific  guidance,  I  would  not  be 
surprised  if  the  majority  of  random  sampling  plans  which  in 
effect  are  of  poor  statistical  validity.  Contractors  have 
had  no  reason  to  challenge  these  procedures  either  because 
they  did  not  understand  them,  or  because  they  were  not  being 
used  as  a  basis  for  deductions  other  than  observed  deficien¬ 
cies.  It  should  be  anticipated  that  the  extrapolation  of 
deductions  will  cause  many  of  these  sampling  plans  to  come 
under  intense  scrutiny. 

Another  impediment  to  the  use  of  random  sampling  has 
been  the  difficulty  of  generating  random  numbers  and  apply¬ 
ing  it  to  the  population.  Appendix  B  contains  instructions 
from  the  MO-327,  which  provides  the  guidance  for  random 
sampling  and  tables  for  random  number  generation.  Sample 
generation  by  the  manual  method  can  be  seen  to  be  time 
consuming  and  difficult  to  apply  objectively.  The  tendency 
would  be  for  users  to  become  frustrated  with  unusable 
numbers  and  just  resort  to  making  up  numbers  on  their  own. 
While  NAVFAC  recognizes  the  shortcomings  of  the  random 
number  table  and  now  advocates  the  use  of  computer  programs 
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for  generating  samples,  they  still  recommend  the  use  of  the 
tables  where  a  computer  and  program  are  not  available. 
[Ref.  19]  I  believe  that  a  computer  based  system  is  a 
necessary  part  of  establishing  a  viable  program  with 
complete  command  support. 

C.  EXTRAPOLATING  DEDUCTIONS 
1.  Background 

NAVFAC  did  not  adapt  the  statistical  methods  of  MIL- 
STED-105D  for  surveillance  of  NAVFAC  service  contracts 
because  it  called  for  acceptance  of  work  at  levels  below 
100%.  Additionally,  it  allowed  large  deviations  in  the 
observed  sample  above  the  AQL  before  it  was  rejected. 
Reservations  about  the  legality  of  extrapolated  deductions 
was  also  expressed.  [Ref.  20] 

In  1983,  NAVFAC  adopted  the  thinking  that  extrapo¬ 
lated  deductions  would  be  an  effective  contractor  motivator. 
It  was  also  thought  that  it  would  provide  an  additional 
incentive  for  activities  faced  with  scarce  inspector 
resources  and  increasing  levels  of  contracted  services  to 
adopt  random  sampling.  The  technique  would  be  implemented 
when  a  methodology  was  available  to  take  deductions  fairly, 
with  a  high  degree  of  confidence  that  they  would  not  take 
the  form  of  punishment.  Dr.  Douglas  Montgomery  of  the 
Georgia  Institute  of  Technology  was  hired  as  a  consultant  to 
develop  a  statistical  basis  and  procedure  for  extracting 
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these  deductions  [Ref.  20],  Working  with  NAVFAC,  a 
methodology  was  ready  for  field  use  in  1984. 

The  ultimate  form  that  extrapolated  deductions  will 
take  is  still  in  doubt.  NAVFAC  is  currently  working  on  a 
new  statistical  basis  which  is  more  in  line  with  its 
contracting  philosophy.  The  elements  of  that  methodology 
will  also  be  looked  at  in  this  chapter. 

2 .  Legal  Considerations 

The  legal  basis  for  extrapolating  deductions  is 

derived  from  the  Federal  Acquisition  Regulations:  [Ref.  21] 

If  any  of  the  services  do  not  conform  with  contract 
requirements,  the  Government  may  require  the  contractor  to 
perform  the  services  again.  .  .  .  When  the  defects  in 

services  cannot  be  corrected  by  reperformance,  the  Govern¬ 
ment  may  .  .  .  (2)  reduce  the  contract  price  to  reflect 

the  reduced  value  of  the  services  performed. 

A  key  concept  is  that  the  reduction  in  price  must 
correspond  to  the  cost  of  the  services.  A  case  that  applied 
this  ruling  was  Environmental  Aseptic  Services  Administra¬ 
tion  and  Larson  Building  Care  Inc.  [Ref.  22].  This  case 
involved  an  Air  Force  janitorial  contract.  The  protesters 
complaint  was  that  sampled  services,  which  in  this  case  was 
"cleaned  rooms,"  subsumed  several  required  tasks  (e.g. , 
aseptic  floor,  furniture,  fixtures,  drapes,  and  trash) .  The 
Performance  Requirements  Summary  provided  that  if  a  task 
fails,  the  room  failed  for  the  day.  In  other  words,  if  the 
contractor  failed  to  clean  the  ashtrays,  he  would  suffer  the 
same  deduction  as  though  he  had  failed  to  perform  all 
required  tasks  in  the  room.  Thus  the  extrapolated  deduction 
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greatly  exceeded  the  value  of  the  one  task  not  performed. 
The  GAO  ruled  in  the  case  that  the  deduction  was  tantamount 
to  a  penalty,  and  its  inclusion  in  the  Invitation  for  Bids 
unnecessarily  raises  the  governments  cost  and  has  an  adverse 
effect  on  competition. 

Closely  tied  to  the  issue  of  reasonable  cost  deduc¬ 
tions  is  the  treatment  of  liquidated  damages.  The  Federal 
Acquisition  Regulations  state  the  following  policy:  [Ref. 
23] 

Liquidated  damages  clauses  should  be  used  when  both  (1) 
the  time  of  delivery  or  performance  is  such  an  important 
factor  .  .  .  that  the  Government  may  reasonably  expect  to 
suffer  damage  if  the  delivery  or  performance  is  delinquent 
and  (2)  the  extent  or  amount  of  such  damage  would  be 
difficult  or  impossible  to  ascertain  or  prove.  .  .  .  The 
rate  of  liquidated  damages  used  must  be  reasonable  and 
considered  on  a  case-by-case  basis.  .  .  .  Liquidated 
damages  fixed  without  any  reference  to  probable  actual 
damages  may  be  held  to  be  a  penalty  and  therefore 
unenforceable . 

The  Navy  Contracting  Manual,  P-68,  allows  liquidated 
damages  to  be  assessed  o  FSCs  at  the  rate  of  10%  of  the 
price  of  the  service  whether  or  not  it  is  subsequently  re¬ 
performed.  The  extrapolation  of  deductions  has  caused  some 
confusion  in  this  area.  Do  liquidated  damages  apply  to  the 
estimated  portion  of  unperformed  work? 

Original  NAVFAC  guidance  indicated  that  the  extrapo¬ 
lation  of  liquidated  damages  was  proper  [Ref.  24].  Later 
guidance  called  for  liquidated  damages  to  be  applied  only  to 
items  re-performed,  which  would  prohibit  their  extrapolation 
[Ref.  19].  More  recent  discussions  with  NAVFAC  indicate 
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that  the  policy  will  be  to  take  liquidated  damages  for 
observed  deficiencies,  whether  or  not  it  is  subsequently  re¬ 
performed  . 

The  key  question  to  be  asked  concerns  the  reason  for 
liquidated  damages.  If  it  is  to  compensate  the  government 
for  increased  administrative  costs  associated  with  re¬ 
inspection,  then  it  is  appropriate  to  apply  liquidated 
damages  only  in  instances  where  the  item  is  re-performed. 
However,  if  the  basis  of  the  liquidated  damages  is  to 
compensate  the  government  for  the  interruption  of  normal 
activities  associated  with  dirty  chalkboards  and  unemptied 
dumpsters,  then  re-performance  is  not  a  consideration  and 
the  damages  remain.  Also,  the  damages  associated  with  the 
estimated  portion  of  nonperformance  would  be  just  as  real  as 
the  inspector  observed  deficiencies.  The  fact  that  NAVFAC 
now  advocates  that  liquidated  damages  be  applied  for 
observed  deficiencies  would  indicate  the  latter  as  a  basis. 
Therefore,  the  extrapolation  of  liquidated  damages  would 
appear  defensible  in  court.  The  greater  amount  of  deduc¬ 
tions  assessed  would  also  act  as  an  enhanced  motivating 
force. 

Another  important  issue  from  a  legal  standpoint  is 
the  implications  of  not  obtaining  a  sample  by  procedure. 
For  example,  suppose  an  inspector  fails  to  obtain  the  sample 
required  by  procedure.  Does  that  invalidate  the  sample  for 
extrapolation  purposes?  From  the  standpoint  of  violating 
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the  procedure  itself,  apparently  not.  The  GAO  has  ruled 
that  when  regulations  set  out  instructions  clearly  for  the 
benefit  of  the  government,  the  agency's  failure  to  comply 
with  it  does  not  provide  a  basis  for  protest.  [Ref.  25] 

As  far  as  the  sample  size  itself,  the  government's 
failure  to  select  an  adequate  sample  has  been  the  basis  for 
upholding  protests  [Ref.  25].  However,  that  case  involved  a 
sampling  scheme  which  was  part  of  the  contract.  NAVFAC  has 
no  plans  to  make  their  quality  assurance  program  a  contract 
requirement.  Even  in  the  case  where  a  sampling  scheme  has 
been  included  as  a  part  of  the  specifications,  the  Armed 
Service  Board  of  Contract  Appeals  has  ruled  that  a  deviation 
in  sampling  procedure  is  allowable  where  "no  appreciable 
difference  in  results  occurs"  [Ref.  26].  I  agree  with  the 
NAVFAC  policy  not  to  include  quality  assurance  procedures  in 
the  contract.  The  government  must  reserve  the  right  to 
remain  flexible  in  this  area.  At  the  same  time,  contrac¬ 
tually  specified  quality  assurance  would  open  up  contract 
administrators  to  numerous  protests. 

D.  STATISTICAL  BASIS  FOR  FIELD  TESTING 
1.  Introduction 

The  statistical  method  devised  by  D.r  Montgomery  for 
use  in  the  NAVFAC  field  tests  are  easily  incorporated  into 
tables  amenable  for  field  use.  These  tables  require  no 
knowledge  of  statistics  for  application.  However,  a  lack  of 
appreciation  for  the  relationships  which  form  the  basis  of 
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the  tables  means  that  the  potential  for  misusing  them 
exists.  Lacking  specific  guidance,  activities  may  unneces¬ 
sarily  increase  their  administrative  burden  in  cases  of 
mismatched  samples  and  populations.  It  should  be  emphasized 
that  the  motivation  behind  extrapolated  deductions  is  to 
better  match  pay  to  services  using  minimal  inspector 
resources.  This  section  looks  at  the  statistical  basis  of 
the  field  tests  and  discusses  the  various  components  upon 
which  the  tables  are  based. 

2 .  Use  of  the  Tables 


Tables  approved  by  NAVFAC  for  use  in  monitoring 
service  contracts  and  taking  extrapolated  deductions  were 
distributed  to  Engineering  Field  Divisions  and  Public  Works 
Centers  in  1984.  Guidance  in  use  of  the  tables  (see  sample 
tables  in  Appendix  C)  was  provided  by  NAVFAC  as  follows: 
[Ref.  24] 

In  order  to  use  the  tables,  the  user  must  first  establish 
the  "Critical  Performance  Level"  (CPL)  which  is  defined  as 
the  level  of  performance  by  the  contractor  that  the 
government  considers  to  be  adequate  and  is  expressed  as  a 
percentage.  The  CPL  is  established  during  the  specifica¬ 
tion  writing  phase  and  should  be  entered  into  the  Perform¬ 
ance  Requirements  Summary.  For  a  given  CPL  and  population 
size,  the  tables  give  the  user  a  sample  size  and  a 
Critical  Number  of  Defects  (CND) .  The  CND  is  the  number 
of  defects  in  the  sample  that  would  signal  poor  contractor 
performance  and  is  the  threshold  for  considering  taking  a 
deduction  based  on  a  statistically  estimated  number  of 
defects.  When  the  number  of  defects  observed  in  the 
sample  equals  or  exceeds  the  CND,  the  number  of  defects  in 
the  population  is  estimated  by  multiplying  the  population 
by  the  percentage  of  defects  found  in  the  sample.  When 
the  number  of  defects  observed  in  the  sample  is  less  than 
the  CND,  deductions  will  be  taken  for  observed  defects 
only.  .  .  .  The  Normal  Surveillance  Table  will  usually  be 
used  for  a  new  contract.  The  Tight  Surveillance  Table 
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should  be  used  if  observed  defects  exceed  two-thirds  of 
the  CND  for  two  consecutive  months.  The  Reduced  Surveil¬ 
lance  Table  should  be  used  if  the  observed  defects  are 
less  than  one-third  of  the  CND  for  two  consecutive  months. 
Normally  when  Reduced  Surveillance  Tables  are  used,  deduc¬ 
tions  will  be  taken  for  observed  defects  only. 

Use  of  the  term  CPL  (equal  to  100-AQL)  was  NAVFAC's 
way  of  stating  that  100%  performance  was  being  contracted 
for,  but  that  a  somewhat  lower  level  would  be  tolerated 
before  adverse  contract  action  would  be  initiated.  The 
inclusion  of  the  CPL  in  the  contract  was  NAVFACs  first 
concession  in  writing  that  less  than  100%  performance  was 
acceptable.  This  change  in  policy  was  made  necessary  by  the 
fact  that  the  CPL  was  the  key  factor  in  taking  extrapolated 
deductions,  and  in  the  course  of  explaining  deductions  to 
the  contractor  it  would  have  become  known  anyway. 

Opinion  is  mixed  whether  or  not  the  CPL  being  part 
of  the  bid  package  would  lead  to  lower  bids  by  contractors 
planning  to  under-perform.  The  incentive  to  do  so  would 
seem  to  be  there.  For  example,  if  the  CPI  was  90%,  a 
contractor  could  cut  his  bid  by  10%  and  plan  to  only  perform 
90%  of  the  work.  If  he  could  maintain  his  actual  perform¬ 
ance  at  90%,  he  could  avoid  extrapolation  95%  of  the  time. 
Although  he  would  have  to  rework  observed  deficiencies  or 
suffer  the  specific  deductions,  these  would  only  be  a  small 
percentage  of  the  actual  deficiencies  which  existed.  The 
result  of  all  this  would  be  a  borderline  contractor  with 
high  administrative  costs.  This  possibility  points  out  a 
weakness  of  including  the  CPL  in  the  contract  under  this 
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type  of  system.  Limiting  the  CPL  to  the  QA  plan  would  allow 
it  to  be  changed  at  a  later  date  if  it  became  necessary. 

It  should  be  noted  that  the  method  approved  is  very 
similar  to  that  used  by  the  Air  Force.  For  a  given  perform¬ 
ance  requirement  and  population  size,  a  CND  (reject  number) 
is  established  at  which  time  extrapolation  can  be  applied. 
The  major  procedural  difference  is  that  below  the  CND  the 
Navy  would  still  deduct  for  observed  defects,  while  the  Air 
Force  did  not.  This  difference  was  due  to  the  Navy's  basic 
requirement  being  100%  performance. 

3 .  Mathematical  Basis  of  Field  Method 

The  mathematical  basis  of  the  tables  used  for  the 
field  test  is  the  normal  approximation  of  the  hypergeometric 
probability  distribution.  The  sample  sizes  shown  in  the 
tables  were  calculated  using  the  relationship  of  equation 
(1)  : 


n 


(CPL) ( 1-CPL) 


(1) 


where: 

n  =  required  sample  size 
CPL  =  critical  performance  level 
N  =  estimated  population  size 
Z  =  normal  one-sided  z-statistic 

W  =  width  of  the  confidence  interval  =  ( 1-CPL) /2 
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The  z-statistic  is  dependent  on  the  confidence  level 
desired.  For  normal  inspection  the  confidence  level  is  95% 
and  the  corresponding  z-statistic  is  1.645.  This  means  that 
the  resulting  sample  will  ensure  that  the  observed  defect 
rate  is  within  the  specified  confidence  interval  95%  of  the 
time.  Tightened  inspection  equals  99%  confidence,  while 
reduced  equals  90%. 

4 .  Effect  of  CPL 

The  confidence  interval  (W)  can  be  seen  to  be  a 
function  of  the  CPL.  It  is  perhaps  easier  to  think  of  it  as 
one-half  the  allowable  defect  rate.  Thus  the  greater  the 
CPL,  the  more  narrowly  defined  is  the  confidence  interval, 
and  the  greater  is  the  sample  size  required.  This  relation¬ 
ship  may  be  undesirable  due  to  the  lack  of  specific  guidance 
in  setting  CPLs.  Contract  administrators  have  interpreted 
the  CPL  to  be  a  level  of  acceptable  performance  from  an 
average  contractor.  Thus  easily  performed  services  should 
rate  a  high  CPL  regardless  of  the  importance  of  the  service. 
Because  the  confidence  interval  then  becomes  such  a  small 
number,  which  is  subsequently  squared,  it  has  an  overriding 
effect  on  the  sample  size.  From  the  normal  table  (see 
Appendix  C)  it  can  be  seen  that  if  the  population  estimate 
is  1000  units  of  work,  a  shift  in  the  CPL  from  90%  to  95% 
would  result  in  the  required  sample  size  being  doubled. 

If  the  confidence  interval  is  going  to  vary 
inversely  with  the  CPL,  a  distinction  should  be  made  between 
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"critical"  services  and  those  which  happen  to  rate  a  high 
CPL  due  to  the  ease  with  which  they  can  be  performed. 
Perhaps  the  latter  category  could  be  inspected  at  a  reduced 
level  of  confidence.  This  would  be  consistent  with  the 
efficient  use  of  inspector  resources. 

The  use  of  the  CPL  term  in  equation  (1)  is  only  a 
best  estimate  of  actual  performance,  which  is  what  is 
actually  mathematically  required  to  make  the  equation  valid. 
If  the  actual  performance  is  less,  the  actual  confidence 
level  achieved  will  be  less.  NAVFACs  rationale  for 
accepting  lower  confidence  levels  when  contractor 
performance  is  below  the  CPL  is  that  greater  attention  will 
be  focused  on  the  contractor's  performance  by  various  other 
methods.  A  plot  of  this  relationship  for  a  95%  confidence 
level  (normal  surveillance)  and  CPLs  of  90%  and  97%  is  shown 
in  Figure  4.1,  a  population  of  300  was  assumed.  It  can  be 
seen  that  achieved  confidence  at  realistic  performance 
levels  still  gives  the  benefit  of  the  doubt  to  the  contrac¬ 
tor.  Therefore,  whether  or  not  other  actions  are  antici¬ 
pated,  very  poor  performance  does  not  invalidate 
extrapolation . 

5 .  Effect  of  Sample  Size 

In  Section  C-2  of  this  chapter,  the  implications  of 
not  taking  the  appropriate  sample  size  was  discussed. 
Figure  4.2  shows  a  plot  of  the  relationship  between  sample 
size  and  confidence  level  for  populations  of  100  and  500, 
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normal  sampling  and  a  CPL  of  90%  are  assumed.  Where  actual 
samples  fall  below  the  required  sample,  achieved  confidence 
gradually  decreases.  This  is  especially  true  where  large 
samples  are  involved.  For  example,  if  the  sample  taken  for 
a  population  of  500  was  only  half  the  required  82,  achieved 
confidence  would  still  be  in  excess  of  86%.  It  can  be 
concluded  that  failure  to  reach  a  designated  sample  is 
probably  not  going  to  appreciably  affect  the  results. 
Therefore  guidance  on  this  matter  should  be  promulgated  to 
avoid  the  situation  of  activities  invalidating  their  samples 
when  there  is  no  need  to  do  so. 

6 .  Effect  of  Population  Estimate 

Similar  to  the  situation  of  not  obtaining  the  proper 
sample  size  is  the  almost  monthly  problem  of  not  properly 
estimating  the  population  size.  Analysis  of  the  relation¬ 
ship  between  confidence  level  and  population  size  shows  that 
this  situation  is  of  almost  no  consequence.  Take  the  case 
where  the  estimated  population  is  500  units  and  the  desired 
confidence  level  is  95%  (CPL  =  90%,  sample  size  =  82).  If 
the  actual  population  turns  out  to  be  400,  the  resulting 
confidence  level  will  be  95.5%.  At  an  actual  population  of 
600,  the  original  sample  of  82  will  still  give  a  confidence 
level  of  94.7%!  In  fact,  whenever  the  original  sample  size 
is  greater  than  about  30,  the  achieved  confidence  level 
should  be  sufficient  to  support  extrapolated  deductions 
using  the  original  CND.  Again,  there  is  no  guidance  on  this 
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matter  and  activities  may  try  to  unnecessarily  match  sample 
sizes  to  ultimate  populations  at  the  expense  of  manpower  and 
possibly  statistical  validity.  It  would  appear  to  be  the 
case  that  in  all  situations  where  random  sampling  is  appro¬ 
priate,  the  original  sample  should  be  sufficient  to  support 
extrapolated  deductions  or  any  other  contract  action. 

7 .  Operating  Characteristics 

The  CND  is  calculated  by  determining  the  defect  rate 
which  exceeds  the  CPL  plus  the  confidence  interval  at  the 
specified  confidence  level.  Equation  (2)  illustrates  this 
relationship: 


CND  =  n  [CPL  +  z  ~\y(CPL)  '(^  CPL)  (1  -^)]  (2) 

The  application  of  equation  (2)  is  equivalent  to  a 
standard  hypothesis  test.  The  statement  being  made  at  a 
normal  level  of  inspection  is  that  if  the  CND  is  reached,  we 
are  95%  confident  that  the  actual  defect  rate  is  equal  to  or 
greater  than  the  allowable  defect  rate. 

Figure  4.3  contains  an  operating  characteristic 
curve  for  various  CPLs.  It  can  be  seen  that  while  this 
methodology  is  biased  toward  the  contractor,  it  does  exhibit 
higher  levels  of  producer  risk  than  MIL-STD-105D.  A  good 
point  of  comparison  is  an  actual  performance  rate  of  85%  and 
a  CPL  of  90%  (AQL  =  10%) .  The  NAVFAC  statistical  basis  will 
accept  this  level  of  performance  just  over  50%  of  the  time. 
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ACTUAL  PERFORMANCE  PERCENT 

Figure  4.3  Characteri sti cs  o-f  Normal  Sampling,  sample=50 
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while  the  Air  Force  would  accept  it  85%  of  the  time  using 
MIL-STD-105D  (see  Appendix  A  for  MIL-STD-105D) . 

E.  PROPOSED  STATISTICAL  BASIS  FOR  FUTURE  USE 

A  problem  with  the  statistical  basis  used  in  the  field 
tests,  as  viewed  by  NAVFAC ,  is  that  it  did  not  allow  for 
extrapolation  until  contractor  performance  was  below  the 
CPL.  This  implied  that  performance  above  the  CPL  was 
acceptable.  Yet  NAVFAC  is  adamant  that  service  contracts  be 
written  for  100%  performance.  Thus,  a  new  method  for  taking 
extrapolated  deductions  is  being  developed  by  NAVFAC.  It  is 
planned  that  when  ready  for  field  use,  this  method  will 
replace  the  method  used  in  the  field  tests.  This  section 
describes  the  characteristics  of  the  new  method  as  it 
currently  exists. 

The  proposed  statistical  basis  for  taking  extrapolated 
deductions  in  the  future  is  more  in  line  with  NAVFAC' s 
policy  of  contracting  for  100%  performance  and  taking 
deductions  where  evidence  of  nonperformance  exists.  Code 
named  RSED  V3 . 0 ,  this  methodology  seeks  to  determine  the 
contractor's  actual  level  of  performance.  Deductions  are 
then  extrapolated  without  regard  to  a  CPL.  The  concept  of  a 
CPL  is  utilized  to  the  extent  that  if  a  contractor's 
performance  does  not  meet  the  CPL,  as  determined  by  the  ODR, 
contract  actions  in  addition  to  payment  deductions  are 
indicated.  These  CPLs  are  to  be  included  in  the  contracts 
Performance  Requirements  Summary.  [Ref.  19] 
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RSED  V3.0  has  not  been  finalized  for  field  use.  Prelim¬ 


inary  indications  are  that  it  will  incorporate  one  basic 
sampling  level,  although  others  may  be  added.  The  defect 
rate  determined  by  the  sample  is  adjusted  downward  by  a 
specified  "delta"  depending  on  the  observed  defect  rate  to 
determine  a  level  appropriate  for  deductions.  The  use  of 
the  delta  ensures  that  the  deduction  rate  will  be  less  than 
or  equal  to  the  contractor's  actual  defect  rate  7  5%  of  the 
time  on  a  monthly  basis,  and  approximately  90%  of  the  time 
over  the  annual  contract  term.  The  higher  confidence  level 
associated  with  the  contract  term  is  felt  by  NAVFAC  to  be 
the  minimum  necessary  for  which  to  base  stronger  contract 
actions  such  as  termination  or  not  renewing  options. 
Appendix  D  contains  the  sample  size  and  delta  tables  along 
with  a  sample  payment  analysis  reproduced  from  NAVFAC 
documentation.  [Ref.  19] 

Like  the  model  used  for  field  testing,  RSED  V3 . 0  uses 
the  normal  approximation  of  the  hypergeometric  distribution 
to  determine  sample  sizes  and  confidence  intervals.  The 
relationship  used  for  calculating  sample  sizes  is  identical 
to  the  field  model  except  that  different  assumptions  are 
made.  Equation  (3)  illustrates: 


n 


w 


(P)  (1-p) 


+ 


(3) 


51 


where : 


n  =  required  sample  size 

W  =  .01  (a  fixed  value  of  1%  regardless  of  CPL) 

P  =  .95  (a  fixed  value  of  anticipated  performance, 

not  CPL) 

Z  =  .674498  (normal  z-statistic  for  75%  certainty) 

N  =  estimated  population  size 

Equation  (3)  can  be  rewritten  as, 

n  =  [.00462749  +  l]"1  (4) 

N 

The  resulting  sample  size  can  be  seen  to  be  independent 
of  the  CPL.  It  will  yield  an  observed  defect  rate  within  1% 
of  the  actual  defect  rate  75%  of  the  time,  provided  the 
actual  performance  is  close  to  the  estimate  of  95%.  At  the 
end  of  the  month,  the  observed  defect  rate  is  used  as  a 
better  estimate  of  actual  performance  to  calculate  a  new 
confidence  interval.  This  new  confidence  interval  is  found 
by  solving  Equation  (1)  for  "W" .  The  new  value  for  "W" 
becomes  the  "delta"  adjustment.  Equation  (5)  illustrates: 

DELTA  =  Z  \JP  (1~P-)-  (i  -g)  (5) 

where : 

P  =  observed  defect  rate  (ODR) 

Z  =  1.28159  (normal  z-statistic  for  95%  certainty) 
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N  =  12,000 
n  =  732 

The  "delta"  is  calculated  with  fixed  values  of  "N"  and 
"n"  so  that  one  table  which  is  a  function  of  the  observed 
defect  rate  may  be  utilized.  This  simplification  results  in 
deltas  approximately  one  percent  less  than  would  have  been 
obtained  using  actual  sample  sizes  and  populations.  It  also 
precludes  the  necessity  of  having  a  table  for  each  possible 
sample  size  and  population!  The  delta  in  conjunction  with 
the  ODR  can  now  be  used  to  estimate  the  true  level  of 
contractor  performance,  and  to  determine  a  deduction  rate  to 
be  used  for  extrapolation.  The  following  statements  help 
summarize  the  results  of  this  methodology  using  an  ODR  of 
10%  as  an  example:  "We  are  75%  confident  that  the  true 
defect  rate  is  greater  than  the  deduction  rate  of  8.62% 
(ODR-DELTA) , :  or  for  evaluating  performance,  "We  are  75% 
confident  that  the  true  performance  level  of  the  contractor 
is  less  than  91.38%  (100%  -  ODR  +  DELTA)." 

F.  COMPARISON  OF  THE  TWO  STATISTICAL  BASES 
1 .  Inspection  Requirements 

The  proposed  statistical  basis  will  require  greater 
sample  sizes  to  be  inspected  than  the  field  version.  This 
is  due  to  the  fact  that  RSED  V3 . 0  seeks  to  ascertain  actual 
contractor  performance  within  a  tolerance  of  one  percent. 
Because  extrapolation  will  be  applied  at  all  levels  of 
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performance  under  RSED  V3.0,  this  small  confidence  interval 
is  deemed  necessary  by  NAVFAC.  Since  the  field  version  is 
designed  only  to  determine  whether  actual  performance  is 
less  than  a  "tolerable"  level,  it  can  be  done  with  a  smaller 
sample  at  a  higher  level  of  confidence. 

Another  difference  between  the  methods  is  the 
greater  flexibility  allowed  with  the  field  version.  Depend¬ 
ing  on  inspector  resources  and  contractor  performance,  SCMs 
can  direct  inspection  at  three  confidence  levels  with  large 
differences  in  inspection  requirements.  In  addition, 
different  CPLs  can  be  assigned  to  further  adjust  inspection 
requirements  within  the  different  levels.  However,  the 
large  influence  on  inspection  requirements  exerted  by  the 
CPL  may  cause  some  contract  administrators  to  specify  a 
performance  level  lower  than  actually  desired.  Under  normal 
levels  of  inspection,  specifying  a  95%  CPL  over  a  90%  CPL 
results  in  almost  doubling  the  sample  size.  The  potential 
thus  exists  for  contract  performance  requirements  to  become 
a  function  of  inspector  resources. 

Increased  flexibility  could  be  built  into  RSED  V3.0. 
This  would  have  to  be  done  by  allowing  different  confidence 
intervals,  since  the  confidence  level  of  75%  would  seem  to 
be  approaching  a  minimum.  Specifying  larger  confidence 
intervals  would  not  change  the  ODR,  which  will  be  the  basis 
for  contract  actions  other  than  extrapolation  and  remains 
the  best  estimate  of  contractor  performance.  Also,  the  fact 
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that  straight  line  deductions  exist  in  the  Air  Force  where 
contract  specifications  have  not  been  met,  implies  that 
NAVFAC  is  being  unnecessarily  conservative. 

2 .  Effect  On  Contractor  Performance 

It  is  reasonable  to  assume  that  the  contractor  will 
be  motivated  closer  to  100%  performance  under  RSED  V3.0. 
However,  most  contract  administrators  feel  that  100% 
performance  is  unrealistic  for  most  services,  and  expressed 
the  opinion  that  public  works  forces  rarely  performed  at  the 
now  specified  CPL.  The  extrapolation  of  deductions  at 
performance  levels  which  have  been  previously  considered 
satisfactory  represent  a  source  of  increased  risk  to  the 
contractor  which  may  be  expected  to  result  in  higher  bid 
prices. 

Bid  prices  may  also  be  higher  under  RSED  V3 . 0  for  a 
more  positive  reason.  The  feature  of  extrapolating  at  all 
defect  levels  means  that  there  is  a  much  closer  match 
between  payment  and  services  received.  Contractors  will  not 
have  any  incentive  to  underbid  the  contract  with  the  goal  of 
performing  at  the  CPL  and  avoiding  extrapolation.  Thus  RSED 
V3 . 0  might  result  in  more  bids  where  the  contractor  is  able 
to  make  a  fair  profit  at  the  required  performance  level. 
Whether  or  not  this  results  in  higher  overall  bids  depends 
on  the  profit  levels  now  being  enjoyed  by  service 
contractors. 
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There  seems  to  be  a  greater  potential  for  an  adver¬ 
sarial  relationship  to  develop  between  the  contract 
administrators  and  contractor  under  RSED  V3.0.  This  is 
because  every  observed  deficiency  would  have  ramifications 
beyond  its  own  merits.  Under  the  field  method,  this  wasn't 
the  case  until  the  CND  was  reached.  At  that  point  there  was 
little  room  for  sympathy  because  the  contractor  was  perform¬ 
ing  at  levels  well  below  specified  levels.  Because  the 
quality  of  services  is  subject  to  some  inspector  judgment, 
borderline  cases  under  RSED  V3 . 0  may  be  dismissed  as 
inspectors  carry  the  increased  burden  of  identifying 
discrepancies  which  will  be  the  basis  for  extrapolation. 

3 .  Deduction  Level 

There  are  too  many  variables  involved  to  try  and 
quantify  the  anticipated  differences  in  deductions  which 
would  be  obtained  using  the  two  statistical  methods.  It 
would  be  fair  to  say  that  the  potential  for  higher  deduc¬ 
tions  exists  under  RSED  V3.0.  The  dichotomy  remains  that 
when  either  system  is  functioning  as  intended,  the  deduc¬ 
tions  will  be  at  a  minimum. 

A  major  difference  between  the  two  methods  is  the 
treatment  of  discrepancies  outside  the  random  sample  (e.g., 
customer  complaints,  planned  sampling) .  When  deductions  are 
extrapolated,  it  is  not  correct  to  also  deduct  for  deficien¬ 
cies  found  outside  the  sample,  as  these  would  be 
theoretically  included  in  the  extrapolated  results.  Since 
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RSED  V3 . 0  extrapolates  from  the  start,  it  is  never 
permissible  to  deduct  for  a  customer  complaint.  The  field 
method  would  allow  this  deduction  if  the  CND  had  not  been 
reached.  An  interesting  result  is  that  in  services  which 
are  customer  oriented  and  a  strong  customer  complaint  pro¬ 
gram  exists,  extrapolated  deductions  under  RSED  V3 . 0  could 
easily  be  less  than  what  would  have  been  obtained  by 
deducting  for  known  deficiencies.  This  does  not  appear  to 
be  a  situation  that  the  contractor  can  take  advantage  of. 
Even  if  the  services  were  not  amenable  to  re-work,  contract 
actions  other  than  extrapolation  could  be  resorted  to. 

4 .  Producer/Consumer  Risk 

Both  the  field  version  and  RSED  V3 . 0  are  biased 
towards  the  contractor.  The  field  version  allows  actual 
defect  rates  to  be  approximately  1.5  times  the  CPL  before  an 
even  chance  of  rejection  occurs  and  straight  line  extrapola¬ 
tion  is  applied.  Performance  levels  under  RSED  V3 . 0  are 
closely  determined,  but  then  a  "delta"  subtracted  from  the 
ODR  assures  that  the  contractor  will  be  overpaid  in  the  long 
run. 

A  normal  approach  to  comparing  the  producer  bias 
inherent  in  any  statistical  techniques  is  the  operating 
characteristic  curve.  However,  since  there  is  no  reject 
number  (CND)  involved  with  RSED  V3.0,  a  curve  for  it  cannot 
be  constructed.  It  is  safe  to  say  that  in  either  method, 
the  contractor  is  treated  more  than  fairly.  A  theoretical 
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even  treatment  would  be  to  apply  the  ODR  as  a  deduction  rate 
at  any  performance  level.  However,  NAVFAC  is  not  willing  to 
live  with  the  idea  that  in  any  one  month,  the  contractor  has 
an  even  chance  of  being  overpaid  or  underpaid,  least  not  at 
reasonable  inspection  levels  (confidence  intervals) . 

5 .  Ease  of  Use 

Both  methods  appear  workable  and  of  comparable 
difficulty  to  implement  and  administer.  However,  under  RSED 
V3.0  required  sample  sizes  are  specified  for  much  narrower 
population  ranges.  This  increases  the  potential  for 
mismatched  samples  with  actual  populations.  If  clear  and 
simple  guidance  isn't  promulgated  which  defines  actions  to 
be  taken  when  procedures  aren't  strictly  followed,  the 
administrative  burden  could  become  excessive. 
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V.  SURVEY  OF  FIELD  TESTS 


A.  INTRODUCTION 

1 .  Purpose 

The  purpose  of  this  survey  is  to  identify  general 
issues  which  should  be  addressed  before  the  extrapolation  of 
deductions  is  approved  for  general  use.  Although  both  of 
the  field  tests  involved  family  housing  maintenance  con¬ 
tracts,  the  focus  of  this  study  remains  oriented  toward 
general  issues.  Preferred  methods  for  applying  extrapolated 
deductions  to  any  specific  contract  type  are  outside  the 
scope  of  this  work. 

2 .  Background 

The  two  activities  involved  in  the  field  test  are 
Public  Works  Center  (PWC)  Great  Lakes  and  PWC  San  Francisco. 
Although  the  housing  maintenance  requirements  of  each 
activity  are  similar,  many  differences  exist  regarding  both 
the  circumstances  of  the  contracting  effort  and  the  approach 
taken  to  employ  extrapolated  deductions.  Key  differences 
include  CPL  requirements,  performance  weighting,  and  the 
method  for  classifying  service  work  for  bid  and  inspection 
purposes . 

Both  maintenance  contracts  were  initially  written  as 
the  basis  for  a  CA  review.  In  the  case  of  San  Francisco, 
the  contract  ran  for  two  years  before  approval  to  use 
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extrapolated  deductions  was  granted  in  July,  1985.  Prior  to 
that  time,  random  sampling  had  been  conducted  using  the 
procedures  of  MO-327. 

At  Great  Lakes,  extrapolated  deductions  was  a  part 
of  the  contract  from  its  inception  in  1984.  NAVFAC  provided 
assistance  in  developing  contract  specifications,  but  the 
QAP  and  related  payment  documents  were  generated  in-house 
from  scratch.  The  methodology  for  taking  deductions  was 
approved  at  the  NAVFAC  level  before  extrapolation  was 
allowed. 

3 .  Data  Collection 

The  collection  of  quantitative  data  was  made  diffi¬ 
cult  due  to  the  great  amount  of  paperwork  which  is  generated 
over  the  course  of  administrating  a  service  contract.  I  was 
limited  to  using  payment  summary  sheets  because  of  the 
fragmentary  nature  with  which  relevant  information  is 
located  within  that  paperwork.  These  summaries  were  not 
directly  comparable,  so  some  information  obtainable  at  one 
activity  could  not  be  collected  at  another.  One  example  was 
an  inability  to  determine  the  percent  of  rework  accomplished 
at  Great  Lakes.  This  prevented  a  deduction  comparison  with 
RSED  V3.0  at  that  activity,  which  was  done  with  the  PWC  San 
Francisco  data. 

Data  collection  was  focused  in  both  cases  on  perform¬ 
ance,  inspection  and  payment  history  of  service  work. 
Service  work  is  normally  customer  generated  and  requires 
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less  than  16  hours  to  complete.  It  constitutes  the  largest 
component  of  work  performed  under  a  housing  maintenance 
contract,  approximately  half.  Random  sampling  was  applied 
to  service  work  for  the  purpose  of  extrapolating  deductions 
at  both  activities.  Other  major  portions  of  housing 
maintenance  includes  change-of-occupancy  work  and  preventive 
maintenance.  Data  on  these  categories  of  work  was  not 
collected  because  the  change-of-occupancy  work  is  100% 
inspected  and  the  preventive  maintenance  data  was  not  easily 
summarized. 

Data  collected  at  PWC  San  Francisco  represented  a 
contract  period  of  11  months.  Data  collected  at  Great  Lakes 
extended  over  a  24  month  period.  In  addition  to  a  collec¬ 
tion  of  historical  data,  interviews  were  conducted  with 
contract  administrators  at  both  activities. 

B.  CONTRACT  REQUIREMENTS 

1 .  Treatment  of  Service  Calls  at  Great  Lakes 

Service  calls  at  Great  Lakes  were  classified  as 

either  emergency,  urgent,  or  routine.  The  implication  for 

the  contractor  was  that  each  required  different  response 

times.  The  definitions  of  each  category  are  given  below 

along  with  the  required  response  time. 

EMERGENCY  CALLS — These  are  problems  which  pose  an  immedi¬ 
ate  danger  to  the  occupants  or  have  the  potential  to  cause 
further  property  damage  (e.g.,  power  lines  down,  overflow¬ 
ing  toilet) .  They  require  contractor  response  within  30 
minutes  on  a  24  hour  basis. 
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URGENT  CALLS — These  are  problems  which  do  not  pose  an 
immediate  threat  to  occupants  or  property,  but  would  soon 
inconvenience  and  affect  the  well-being  of  the  occupants 
(e.g. ,  refrigerator  not  working,  toilet  backed  up)  .  The 
contractor  is  required  to  respond  within  two  hours  of  the 
call . 

ROUTINE  CALLS — These  are  problems  which  cannot  be  classi¬ 
fied  as  either  an  emergency  or  urgent  situation  (e.g., 
tile  repair,  binding  doors) .  The  contractor  is  required 
to  respond  within  nine  days  and  to  complete  the  work 
within  14  days. 

The  performance  of  service  calls  was  broken  down 
into  four  performance  indicators:  responsiveness,  quality 
of  work,  proper  classification,  and  proper  accounting.  Each 
one  of  the  performance  indicators  was  assigned  a  weighting 
representative  of  the  "worth"  of  that  portion  of  the  job. 
Performance  weights  must  total  100%.  When  performance 
indicators  are  weighted,  it  means  that  accomplishment  of  any 
individual  performance  indicator  earns  a  percent  of  the 
service  price  equal  to  the  weighting.  Thus,  weighting  has  a 
direct  bearing  on  the  level  of  deductions  to  be  assessed  for 
failure  to  perform.  For  example,  the  performance  weighting 
for  emergency  response  is  45%.  Suppose  that  a  contractor 
responded  to  an  emergency  call  one  hour  late.  Regardless  of 
the  fact  that  quality  work  was  done  and  the  job  was  properly 
documented,  45%  of  the  price  would  be  subject  to  deduction. 

2 .  Treatment  of  Service  Calls  at  San  Francisco 


Service  calls  at  San  Francisco  were  also  categorized 
as  emergency,  urgent  and  routine.  However  the  required 
response  times  differed  as  follows: 
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EMERGENCY  CALLS — two  hours  during  work  hours,  four  after; 
URGENT  CALLS — twenty  four  hours; 

ROUTINE  CALLS — five  days  response,  fourteen  days 
completion. 

San  Francisco  also  used  the  same  four  performance 
indicators  as  Great  Lakes,  but  did  not  assign  weights  to 
them.  In  effect,  that  meant  that  each  one  was  weighted 
100%.  A  deficiency  in  response  time,  quality,  or  documenta¬ 
tion  would  result  in  a  total  deduction  being  taken. 

Another  difference  between  the  activities  was  that 
San  Francisco  combined  its  service  calls  into  three  other 
categories  for  the  purpose  of  bidding  and  inspection.  These 
were  interior,  exterior  and  appliances.  Thus,  exterior  work 
might  consist  of  any  of  the  three  service  call  types. 

C.  QUALITY  ASSURANCE  PLANS 

1 .  Introduction 

Quality  Assurance  Plans  (QAP)  are  long,  complex 
documents  which  attempt  to  provide  the  activity  with  a 
structured  approach  for  inspecting  all  aspects  of  a  contrac¬ 
tor's  performance.  It  is  beyond  the  scope  of  this  paper  to 
attempt  to  convey  the  contents  of  either  activity's  QAP.  My 
intent  is  to  report  only  those  findings  which  have  general 
applicability  to  RSED. 

2 .  General  Comments 

A  major  difference  between  the  two  plans  was  that 
PWC  Great  Lakes  wrote  theirs  initially  to  incorporate  RESED. 


63 


San  Francisco's  plan  was  based  on  the  MO-327  version  of 
random  sampling,  which  did  not  allow  for  extrapolation.  The 
result  was  that  the  QAP  in  San  Francisco  had  little 
resemblance  to  the  quality  assurance  program  which 
eventually  evolved.  The  fact  that  PWC  San  Francisco  was 
able  to  implement  a  workable  program  after  some  trial  and 
error,  reinforces  the  wisdom  of  separating  the  QAP  from  the 
formal  contract. 

3 .  Critical  Performance  Levels 

The  CPLs  contained  in  the  QAP  were  not  part  of  the 
bid  information  of  either  contract.  NAVFAC's  policy  then, 
which  has  just  recently  changed,  is  that  the  CPL  was  an  in- 
house  tool  only.  However,  while  unknown  to  the  contractor 
during  the  bidding  process,  the  contractor  soon  became 
familiar  with  the  CPL  as  it  formed  the  basis  of  payment 
analysis  and  performance  judgments. 

The  CPL  assigned  to  service  contracts  in  San 
Francisco  was  90%,  while  Great  Lakes  enforced  a  CPL  of  97%. 
San  Francisco  felt  that  a  CPL  in  excess  of  90%  was  too  harsh 
and  unrealistic.  The  opinion  at  Great  Lakes  was  that  97% 
performance  was  realistic  and  had  been  achieved  by  the 
departing  public  works  force.  The  SCM  at  Great  Lakes  also 
felt  that  a  97%  CPL  was  high  enough  to  prevent  planned 
noncompliance  by  the  contractor.  A  possibility  he  thought 
plausible  at  a  CPL  level  of  90%. 
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The  actual  weighted  average  performance  or  service 
calls  achieved  at  PWC  San  Francisco  was  92.0%,  as  compared 
to  98.1%  at  PWC  Great  Lakes.  While  contractor  performance 
is  subject  to  many  variables,  including  inspector 
aggressiveness,  the  close  parallel  between  performance 
differences  and  CPL  differences  suggests  a  relationship. 

4 .  Performance  Weighting 

The  failure  of  PWC  San  Francisco  to  weight  the 
different  aspects  of  service  call  performance  for  payment 
purposes  would  probably  have  made  extrapolation  unsupporta- 
ble  in  court.  This  is  similar  to  the  Air  Force  janitorial 
contract  [REf.  2]]  where  a  room  could  be  declared  unsatis¬ 
factory  and  payment  denied  because  an  ashtray  was  not 
emptied. 

Not  weighting  performance  factors  also  makes  bidding 
more  difficult  and  transfers  greater  risk  to  the  contractor. 
This  may  have  been  a  factor  in  PWC  San  Francisco  paying  a 
weighted  average  of  $55.00  per  service  call,  while  Great 
Lakes  paid  only  $33.00. 

There  was  not  enough  data  to  support  conclusively 
whether  or  not  weighting  influenced  contractor  performance 
at  Great  Lakes  (see  performance  weights  in  Figure  5.1). 
However,  in  emergency  and  urgent  calls  where  responsiveness 
and  quality  were  closely  weighted,  mean  performance  was 
within  one  half  of  a  percent  of  each  other.  In  the  case  of 
routine  calls,  where  quality  carried  twice  the  weight  of 
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responsiveness,  quality  performance  exceeded  responsiveness 


by  4.3%. 

D.  INSPECTION  REQUIREMENTS 

1.  Introduction 

This  section  looks  briefly  at  the  use  of  inspector 
resources  at  each  activity.  Any  random  sampling  program 
approved  for  extrapolated  deductions  must  permit  efficient 
use  of  inspectors,  with  minimal  administrative  burden,  if  it 
is  to  receive  command  support.  Although  many  of  the  inspec¬ 
tion  procedures  referred  to  are  in  the  QAP,  all  inspection 
related  information  is  presented  in  this  section  for 
clarity. 

2 .  Sampling  Levels 

Figures  5.2  and  5.3  show  the  monthly  level  of 
inspection  employed  at  each  activity  over  the  period  studied 
along  with  the  population  range  for  each  of  the  service 
categories.  It  should  be  noted  that  due  to  the  lower  CPL, 
inspection  levels  at  San  Francisco  were  far  below  that  of 
Great  lakes.  In  fact,  although  the  number  of  calls  at  San 
Francisco  are  higher,  they  inspected  less  than  half  the 
calls  inspected  at  Great  Lakes.  This  relationship  would 
suggest  too  great  an  incentive  for  contract  administrators 
to  sacrifice  performance  requirements  for  the  sake  of  saving 
inspector  resources. 

These  figures  also  show  the  potential  savings 
possible  by  combining  the  services  into  one  group  for 
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POPULATION 

MEAN 

POPULATION 

RANGE 

SAMPLE 

MEAN 

INSPECTION 

LEVEL 

ROUTINE  CALLS 

536 

391-713 

207 

397. 

URGENT  CALLS 

470 

381-604 

163 

357. 

EMERGENCY  CALLS 

165 

110-250 

97 

597. 

TOTAL 

1171 

950-1391 

467 

407. 

Figure  5.2 

Inspect i on 

Data  at  PWC 

Great  Lakes 

POPULATION 

MEAN 

POPULATION 

RANGE 

SAMPLE 

MEAN 

INSPECTION 

LEVEL 

INTERIOR  CALLS 

961 

814-1070 

88 

97. 

EXTERIOR  CALLS 

280 

173-435 

69 

257. 

APPLIANCE  CALLS 

192 

139-229 

64 

347. 

TOTAL 

1433 

1252-1669 

22 1 

157. 

Figure  5.3  Inspection  Data  at  PWC  San  Francisco 
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inspection  purposes.  If  the  population  of  service  calls  at 
San  Francisco  were  combined  into  one  group  of  14  3  3  calls, 
the  required  sample  size  would  only  be  91!.  Whether  or  not 
this  could  reasonably  be  considered  a  homogeneous  population 
is  subject  to  debate.  Differences  in  frequency  or  cost  is 
no  greater  than  the  variation  which  occurs  within  each 
service  call  category.  The  economy  inherent  in  large 
populations  argues  for  combining  work  items  where  possible. 

A  look  at  the  population  ranges  in  Figures  5.2  and 
5.3  illustrates  the  difficulty  in  estimating  an  appropriate 
sample  size.  Another  benefit  to  increasing  the  population 
is  the  greater  stability  involved.  A  sample  size  of  91 
would  apply  to  all  populations  in  the  range  of  1300-1500. 

3 .  Surveillance  Level  Switching 

Both  QAPs  provided  conditions  under  which  surveil¬ 
lance  levels  could  be  changed.  At  PWC  San  Francisco,  the 
option  was  not  exercised  for  two  reasons:  (1)  contract 
administrators  felt  that  reduced  inspection  levels  would  not 
have  been  adequate,  and  (2)  tight  surveillance  was  beyond 
the  capacity  of  the  inspector  force. 

Surveillance  levels  were  switched  as  deemed  appro¬ 
priate  by  the  SCM  at  PWC  Great  Lakes.  Although  the  guidance 
in  the  QAP  was  not  strictly  followed,  the  switching  tech¬ 
nique  was  effectively  utilized.  Even  within  the  routine 
service  work  category,  different  surveillance  levels  were 
used  for  a  period  of  time.  Response  time  was  under  tight 
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inspection,  while  quality  of  work  was  inspected  under 
reduced  surveillance.  It  should  be  remembered  that  the 
incentive  for  going  to  reduced  inspection  was  much  greater 
at  Great  Lakes.  Even  under  reduced  inspection,  at  a  97%  CPL 
level,  greater  inspection  was  required  than  normal  inspec¬ 
tion  at  San  Francisco. 

The  switching  of  inspection  levels  does  present  a 
problem  in  balancing  the  workload.  The  SCM  at  Great  Lakes 
claimed  that  he  was  staffed  for  a  normal  inspection  load. 
When  tight  inspection  was  implemented,  as  it  was  for  the 
first  year  under  routine  response,  it  could  be  covered  only 
through  the  use  of  overtime  and  the  shifting  of  inspectors 
from  other  assignments.  In  my  view,  the  only  thing  gained 
by  tightened  inspection  is  the  ability  to  extrapolate  the 
same  percentages  at  a  higher  confidence  level  and  a  few  more 
observed  deficiencies.  It  would  seem  more  efficient  to  stay 
at  normal  inspection  for  extrapolation  purposes  and  use 
planned  inspections  where  it  was  necessary  to  prod  the 
contractor  a  little  further.  Planned  inspections  on  a 
limited  basis  can  usually  be  done  so  that  the  incidence  of 
deficiencies  is  enhanced.  Also,  these  extra  planned 
inspections  would  not  be  a  firm  requirement  and  would  thus 
allow  extra  flexibility.  While  deductions  resulting  from 
planned  inspections  are  allowed  only  if  extrapolation  does 
not  take  place,  the  rework  requirement  is  still  enforceable. 
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4 .  Sample  Generation 


Both  activities  made  use  of  a  computer  program  to 
generate  random  samples.  At  the  beginning  of  the  month,  the 
computer  would  generate  the  random  numbers  and  these  would 
be  matched  to  service  call  chits  as  they  came  in. 

The  better  program  was  in  use  at  Great  Lakes  in  that 
it  did  not  require  the  use  of  tables  (see  tables  in  Appendix 
C) .  Required  input  consisted  of  a  population  estimate,  CPL, 
surveillance  level  and  lowest  sample  number.  At  San 
Francisco  only  the  population  estimate  and  sample  size  was 
required  for  input,  but  determining  the  appropriate  sample 
size  required  using  the  tables.  Aside  from  the  enhanced 
possibility  of  making  a  mistake,  the  stratification  of  the 
tables  normally  meant  that  required  samples  were  slightly 
greater  than  if  determined  directly  by  formula.  In 
addition,  the  San  Francisco  program  always  generated  numbers 
starting  from  one,  which  meant  that  chits  needed  to  be 
renumbered  every  month. 

5 .  Sample  Adjustment 

The  question  of  sample  adjustment  arises  when  the 
actual  population  of  services  in  a  month  requires  a 
different  sample  than  the  one  obtained  as  a  result  of  the 
prior  estimate  of  the  population.  At  PWC  San  Francisco, 
this  possibility  was  not  addressed  in  the  QAP.  Lacking 
clear  guidance,  command  procedure  has  been  to  evaluate  the 
sample  as  if  the  population  estimate  had  been  correct.  This 
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is  a  simplifying  procedure  based  on  conserving  inspector 
resources.  In  evaluating  the  sampling  history,  this 
experience  has  been  justified.  Due  to  the  relatively  large 
samples  involved,  the  lowest  being  appliances  at  a  mean  of 
64  inspections  per  month,  little  change  in  confidence  levels 
would  have  been  gained  by  doing  additional  sampling.  There 
was  not  time  where  an  adjustment  could  have  resulted  in 
extrapolation. 

The  procedure  written  for  PWC  Great  Lakes  was  more 
sophisticated,  yet  was  unnecessary.  At  the  end  of  the 
month,  the  QAE  adjusted  the  sample  size  using  the  computer 
program.  By  giving  it  the  same  "seed"  number,  it 
automatically  generated  a  new  sample  which  either  adds  a  few 
numbers  to  the  previous  sample  or  deletes  a  few.  The  QAE 
would  then  either  inspect  the  additional  samples  or  invali¬ 
date  some  already  inspected  for  purposes  of  extrapolation. 
It  is  never  desirable  to  invalidate  a  sample,  as  this  only 
reduces  the  confidence  level  of  the  results. 

As  practiced,  the  SCM  decides  whether  or  not  to  do 
sample  adjustment  based  on  the  possibility  of  extrapolation. 
Where  the  number  of  defects  is  close  to  the  CND,  he  will 
authorize  a  sample  adjustment.  Even  this  procedure  is  not 
always  prudent.  Suppose  the  estimated  population  for 
emergency  calls  is  175.  At  a  CPI  of  97%  and  normal  surveil¬ 
lance,  this  would  call  for  a  sample  size  of  117  and  a  CND  of 
six.  If  the  population  turned  out  to  be  200,  it  would  call 
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for  a  sample  size  of  127  and  a  CND  of  seven.  Assuming  the 
original  number  of  defects  found  was  six,  the  inspector 
would  be  sent  out  to  sample  ten  more  jobs  in  search  of  one 
more  defect.  However,  had  extrapolation  been  assessed  on 
the  basis  of  the  original  sample,  the  unsatisfactory  deter¬ 
mination  would  have  been  made  at  a  confidence  level  of  93%. 
Certainly  this  is  good  enough  to  ascertain  unsatisfactory 
performance.  Assuming  that  the  original  estimate  had  been 
made  in  good  faith,  this  extrapolation  would  no  doubt  have 
been  upheld  in  court.  It  would  appear  that  guidance  is 
necessary  in  this  area,  so  that  activities  can  make  the  most 
intelligent  use  of  their  resources. 

E.  PAYMENT  ANALYSIS 

1 .  Introduction 

Payment  analysis  at  both  activities  was  based  on 
general  NAVFAC  guidance  issued  early  in  1984  [Ref.  24]. 
Each  activity  modified  the  guidance  slightly  to  make  it  more 
applicable  to  payment  for  service  call  work.  This  section 
evaluates  the  methodology  of  taking  extrapolated  deductions 
at  each  activity.  The  deductions  which  resulted  are  then 
analyzed  in  light  of  the  performance  realized. 

2 .  Payment  Methodology 

Figure  5.4  is  an  example  of  payment  analysis  for 
emergency  service  calls  at  Great  Lakes.  The  procedure 
followed  at  San  Francisco  is  illustrated  by  a  reproduction 
of  the  payment  calculation  for  exterior  maintenance  in 
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October,  1985.  This  calculation  is  shown  in  Figure  5.5. 
The  obvious  difference  between  the  two  is  that  performance 
weighting  requires  separate  analysis  for  each  performance 
indicator  at  Great  Lakes. 

The  wording  of  line  "I"  instructions  (see  Figures 
5.4  and  5.5)  was  found  to  have  caused  problems  at  both 
activities.  Line  ,,I",  as  used  at  Great  Lakes,  is  identical 
to  NAVFAC  guidance.  However,  the  confusing  instructions 
caused  an  inspector  to  not  extrapolate  deductions  when  the 
observed  defects  equaled  the  CND. 

The  instructions  associated  with  line  "I"  at  San 
Francisco  leaves  no  clue  as  to  what  the  actions  should  be 
when  the  observed  defects  equal  the  CND.  In  the  one 
instance  where  the  situation  occurred,  deductions  were  not 
extrapolated  when  the  observed  defects  equaled  the  CND. 
Contract  administrators  were  unaware  of  the  proper  procedure 
to  follow  in  this  case.  Clearer  instructions  should  advo¬ 
cate  extrapolation  when  the,  "observe  defects  are  equal  to 
or  greater  than  the  CND." 

3 .  Treatment  of  Liquidated  Damages 

Liquidated  damages  were  extrapolated  at  both 
activities  (see  Figures  5.4  and  5.5).  The  SCM  at  Great 
Lakes  justifies  this  action  by  claiming  that  the  customer 
suffers  damages  through  the  failure  of  the  contractor  to 
perform.  This  issue  remains  unresolved.  In  their  new 
contracts,  San  Francisco  will  continue  to  extrapolate 
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EXTERIOR  SERVICE  CALL  DEDUCTIONS 

A. 

COST  OF  REQUIRED  SERVICES 

$17, 809.09 

B. 

TOTAL  UNITS  OF  WORK 

262 

C. 

COST  PER  UNIT  OF  WORK 

$64. 94 

D. 

CRITICAL  PERFORMANCE  LEVEL  (CPL) 

907. 

E. 

CRITICAL  NUMBER  OF  DEFECTS 

12 

F. 

SAMPLE  SIZE 

72 

G. 

NUMBER  OF  OBSERVED  DEFECTS 

15 

H. 

PERCENT  OF  DEFECTIVE  UNITS  IN  SAMPLE 

20 . 37. 

I. 

DEDUCTION  FOR  DEFECTIVE  UNITS 

(1)  If  G  is  less  than  E  (GXC) 

NA 

(2)  If  G  is  more  than  E  (HXA ) 

$3. 704.29 

J. 

DEDUCTION  FOR  LIQUIDATED  DAMAGES  (1X107.) 

$370.43 

K. 

NUMBER  OF  RANDOM  SAMPLE  UNITS  REWORKED 

15 

L. 

PAYMENT  FOR  REWORK  (KXC) 

$974. 10 

M. 

NUMBER  OF  VALIDATED  CUSTOMER  COMPLAINTS 

7 

N. 

COST  OF  CUSTOMER  COMPLAINTS  (MXC) 

$454. 58 

0. 

LIQUIDATED  DAMAGES  FOR  CUST  COMP  (NX  107.) 

$45. 46 

P. 

CUSTOMER  COMPLAINTS  REWORKED 

Q. 

PAYMENT  FOR  REWORK 

$129.88 

R. 

DEDUCTIONS  FOR  CUST  COMPLAINTS  (N-O-Q) 

$370. 16 

S. 

TOTAL  DEDUCTIONS  (I+J+O+R) 

$4. 444. 88 

T. 

PAYMENT  FOR  SERVICES  (A-S) 

$13.364. 21 

Figure  5.5  Exterior  Service  Payment  at  F'WC  San  Francisco 
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liquidated  damages,  while  Great  Lakes  will  claim  liquidated 
damages  only  for  work  which  is  re-inspected. 


A  firm  policy  needs  to  be  established  for  each 
category  of  work.  Where  a  government  function  is  the  recip¬ 
ient  of  the  service,  as  in  a  janitorial  contract,  extrapo¬ 
lated  damages  may  be  more  easily  supported.  This  is  a 
different  situation  than  attempting  to  assess  damages  for 
the  housing  occupants'  inconvenience.  In  any  case,  the 
extrapolation  of  liquidated  damages  can  be  used  as  extra 
incentive  to  obtain  contractor  compliance  with  the  contract. 

4 .  Treatment  of  Validated  Customer  Complaints 

Both  activities  took  deductions  for  validated 
customer  complaints  in  addition  to  extrapolation.  This 
procedure  is  incorrect.  The  extrapolation  of  deductions 
theoretically  accounts  for  all  deficiencies  which  exist 
outside  the  sample.  The  most  advantageous  way  to  treat 
validated  customer  complains  under  the  field  version  is  to 
take  deductions  unless  the  CND  has  been  reached.  As  can  be 
seen  in  Figure  5.5,  PWC  San  Francisco  makes  no  such 
distinction. 

5 .  Deduction  Data 

Extrapolation  occurred  ten  times  at  PWC  Great  Lakes 
and  only  once  at  PWC  San  Francisco,  although  each  lost  an 
additional  opportunity  to  extrapolate  when  the  observed 
defects  equaled  the  CND.  The  difference  can  be  attributed 
mainly  to  the  lower  CPL  used  at  San  Francisco.  However, 
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it's  the  potential  of  extrapolated  deductions  that  is  being 
counted  on  to  improve  contractor  performance.  Greater 
deductions  are  not  being  sought  for  their  own  sake.  This 
potential  is  evident  by  analyzing  the  nature  of  the 
deductions . 

A  total  of  $19,195.00  was  taken  in  service  call 
deductions  over  the  24  months  evaluated.  That  represents 
2.1%  of  the  contracted  price  for  these  services.  Forty-five 
percent  of  those  deductions,  $8,694.00,  were  attributable  to 
extrapolation  beyond  observed  defects. 

While  only  one  incident  of  extrapolation  took  place 
in  eleven  months  at  San  Francisco,  it  resulted  in  deductions 
of  $2,731.00  above  the  observed  deductions.  That  represent¬ 
ed  23%  of  the  total  deductions  of  $11,768.00.  Total 
deductions  at  San  Francisco  amounted  to  a  reduction  in  the 
contract  price  of  1.3%. 

Thus  it  can  be  seen  that  a  contractor  certainly  has 
reason  to  avoid  extrapolation.  When  only  a  small  portion  of 
a  contract  is  being  inspected  the  number  of  observed  defects 
will  also  be  small.  Therefore,  deductions  taken  for 
observed  defects  will  be  minor.  Where  the  contractor  can 
also  expect  to  rework  the  item,  there  is  little  financial 
incentive  to  do  it  right  the  first  time.  However,  if 
deductions  are  extrapolated,  the  final  percent  deducted  will 
more  closely  reflect  actual  performance.  The  impact  of  this 
is  shown  graphically  in  the  next  section. 
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6. 


Effect  on  Performance 


It  is  impossible  to  definitively  state  what  factors 
influenced  performance  at  each  activity.  One  fact  is  that 
both  contractors  quickly  learned  to  avoid  extrapolation. 
The  ten  incidences  of  extrapolation  at  PWC  Great  lakes 
occurred  in  the  first  eleven  months  of  the  contract.  Seven 
of  those  extrapolations  were  assessed  for  unsatisfactory 
performance  in  response  to  routine  calls.  Response  to 
routine  calls  was  the  only  area  where  Great  Lakes  experi¬ 
enced  any  problems. 

Figure  5.6  plots  contractor  performance  in  response 
to  routine  calls  at  Great  Lakes,  along  with  the  correspond¬ 
ing  percent  of  contract  payment  each  month.  Extrapolation 
took  place  from  June  to  October;  and  then  again  in  December 
and  February.  It  should  be  noted  that  in  these  months 
payment  actually  fell  below  the  performance  level.  This  is 
an  undesirable  state  of  affairs  from  NAVFAC's  point  of  view, 
and  occurred  because  deductions  were  taken  for  validated 
customer  complaints  in  addition  to  the  extrapolation.  Due 
to  the  high  CPL  of  97%,  the  contractor  was  forced  to  perform 
within  a  relatively  narrow  band  to  avoid  extrapolation.  In 
February  of  1985,  for  example,  the  CND  was  exceeded  even 
though  performance  was  at  95.4%.  The  last  year  of  the  con¬ 
tract  has  seen  steady  high  performance  in  all  service  areas. 

Performance  at  PWC  San  Francisco  was  erratic  in  all 
service  categories.  Figure  5.7  plots  performance  in 
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MOUTHS 


Figure  3.6  Perf ormanca/Payment  Comparison  at  PWC  Great  Lakes 


Figure  5.7  Perf ormanca/Payment  Comparison  at  PWC  San  Francisco 
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exterior  maintenance,  along  with  the  corresponding  percent 
of  contract  payment  each  month.  Except  for  the  extrapola¬ 
tion  which  occurred  in  the  first  month,  performance  in 
exterior  maintenance  typified  the  other  two  work  categories. 
The  potential  of  extrapolated  deductions  is  most  evident 
from  this  plot.  It  can  be  seen  that  where  extrapolation 
took  place,  payment  was  well  below  performance.  Again,  this 
is  a  result  of  taking  deductions  outside  the  sample. 
However,  due  to  the  low  CPL  of  90%,  the  contractor  was  able 
to  perform  at  levels  as  low  as  85.7%  without  exceeding  the 
CND.  Because  the  level  of  inspection  was  low,  large  gaps 
existed  between  payment  and  performance.  In  fact,  the 
closer  the  contractor  could  get  to  the  CND  without  reaching 
it,  the  greater  would  be  his  favorable  gap  between  perform¬ 
ance  and  payment.  Thus,  it  appears  that  the  contractor  was 
able  to  manipulate  his  performance  to  maximize  payment 
differential . 

7 .  Comparison  with  RSED  V3 . 0 

I  did  a  deduction  analysis  with  the  PWC  San 
Francisco  data  assuming  that  RSED  V3 . 0  had  been  in  affect. 
The  questionable  assumption  in  such  an  analysis  was  that 
contractor  performance  remained  the  same.  This  is  question¬ 
able  because  performance  would  have  been  judged  unsatsifac- 
tory  eight  times  vice  once,  resulting  in  more  pressure 
through  CDRs.  In  addition,  the  fact  that  payment  would  have 
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more  closely  matched  performance  would  have  provided 
additional  incentive. 

The  level  of  deductions  under  RSED  V3 . 0  would  have 
been  $3,766.00  per  month.  This  represents  4.7%  of  the 
contract  price  and  a  monthly  increase  in  total  deductions 
over  the  field  method  of  $2,696.00.  This  increased  deduc¬ 
tion  level  would  have  to  be  paid  for  with  increased  inspec¬ 
tions.  Monthly  inspections  required  by  RSED  V3 . 0  would 
represent  an  81%  increase  from  221  inspections  to  401.  At 
least  one  new  inspector  would  have  been  required  to  cover 
this  requirement.  The  extra  deductions  would  pay  for  that 
new  inspector  only  if  performance  did  not  increase! 

Discrepancies  between  pay  and  performance  will  be 
minimal  at  all  levels  under  RSED  V3.0.  Payment  will 
normally  be  about  1%  in  excess  of  performance  if  this 
methodology  is  properly  applied.  This  will  eliminate  any 
motivation  for  the  contractor  to  underperform,  as  now  exists 
at  San  Francisco.  However,  the  fact  that  all  deductions 
will  be  extrapolated  removes  a  clear  goal  tied  to  the 
pocketbook.  That  "goal"  appeared  effective  in  ensuring 
contractor  performance  within  a  narrow  band  at  PWC  Great 
Lakes . 
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VI .  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  GENERAL  ISSUES 

1 .  Need  for  Oversight 

This  study  found  that  in  a  few  areas,  discrepancies 
existed  between  the  RSED  procedures  implemented  at  the  test 
activities  and  the  intentions  of  NAVFAC.  These  discrepan¬ 
cies  occurred  because  of  either  a  lack  of  clear  guidance  or 
an  omission  on  the  part  of  the  activity.  Major  examples 
indicating  a  lack  of  communication  between  NAVFAC  and  the 
activities  are:  (1)  the  failure  of  both  activities  to 
extrapolate  when  observed  defects  equaled  the  CND,  (2)  the 
taking  of  deductions  in  addition  to  extrapolation,  and  (3) 
different  approaches  planned  for  the  extrapolation  of 
liquidated  damages. 

Problems  such  as  these  must  not  only  be  anticipated 
in  future  guidance,  but  incorporated  into  sample  working 
documents  where  possible.  In  addition,  the  potential  for 
misunderstandings  justifies  an  internal  command  assistance 
program.  Evaluation  of  RSED  plans  at  the  EFD  level  would 
promote  uniformity  and  provide  assistance.  This  simple 
followup  would  help  preclude  the  Air  Force  experience  with 
GAO  audits.  Because  of  the  small  number  of  activities 
involved,  effective  oversight  would  not  be  difficult  to 
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implement.  When  correct  procedures  became  entrenched  in  the 
system,  this  special  effort  could  be  phased  out. 

2 .  Treatment  of  Sample  Adjustment 

The  review  of  activity  QA  plans  and  discussion  with 
contract  administrators  indicated  that  the  potential  for 
unnecessary  sample  adjustment  was  great.  Current  guidance 
does  not  clearly  address  the  issue  of  sample  size  adjust¬ 
ment.  This  problem  becomes  even  more  critical  under  RSED 
V3 . 0  due  to  the  monthly  extrapolations  and  narrow  sampling 
requirements.  Lacking  statistical  backgrounds,  contract 
administrators'  actions  are  unpredictable.  Worse  than  the 
expedient  procedure  at  PWC  San  Francisco,  would  be  to 

overcompensate . 

The  analysis  of  the  effect  that  sample  size  had  on 
confidence  level  suggests  that  original  samples  should 
provide  adequate  basis  for  contract  action  in  all  but  the 
most  extreme  cases.  A  simple  cutoff  might  be  a  minimum 

sample  size  of  50.  Where  the  original  sample  is  based  on  a 
good  faith  estimate  and  exceeds  50,  no  adjustment  would  be 
necessary.  Specifying  a  minimum  confidence  level  of  50% 
before  adjustment  was  necessary  would  be  another  approach. 
However,  that  would  require  software  support  to  speed  the 
process  and  ensure  against  miscalculation. 

3 .  Combining  Homogeneous  Populations 

The  benefits  of  random  sampling  increase  exponen¬ 
tially  with  population.  Due  to  the  nature  of  service 
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contracting,  strict  homogeneity  is  almost  never  achieved 
within  any  population.  Where  two  services  have  the  same  CPL 
and  similar  unity  costs,  the  possibility  of  combining  those 
populations  should  be  examined.  Inspection  to  different 
standards  would  not  present  a  problem.  This  process  would 
be  similar  to  the  combination  of  service  calls  into  another 
category  as  was  done  at  San  Francisco.  However,  in  that 
case  no  benefits  were  realized  because  three  categories  were 
still  used.  Candidates  for  combination  should  be  studied 
and  methods  promulgated  to  make  RSED  more  efficient. 

4 .  Surveillance  Levels 

The  wisdom  of  sampling  at  increased  levels  of 
inspection  is  questionable.  Tightened  inspection  does  not 
result  in  increased  levels  of  extrapolation.  Sample  sizes 
required  by  tightened  inspection  (99%  confidence)  are  signi¬ 
ficantly  higher  than  normal  sampling.  It  was  done  at  PWC 
Great  lakes  through  the  use  of  overtime  and  reassignment. 
At  PWC  San  Francisco,  constrained  resources  prevented  its 
use. 

Where  contractor  performance  has  been  poor  and 
inspector  resources  are  available,  minimum  levels  of  random 
sampling  should  be  supplemented  with  planned  sampling.  This 
procedure  makes  use  of  inspector  judgment  to  enhance  the 
probability  of  identifying  defects  while  maintaining 
extrapolation.  More  flexibility  is  gained  because  there  is 
no  need  to  obtain  a  "proper"  sample  at  the  higher  level. 
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Future  guidance  should  clearly  document  the  procedures  for 
mixing  these  two  QA  techniques  and  promote  its  use. 

5 .  Performance/Pavment  Match 

Under  the  field  version,  the  mismatch  between 
payment  and  performance  increases  with  a  lower  CPL.  While 
this  mismatch  provides  a  powerful  incentive  to  avoid 
extrapolation,  it  appeared  to  result  in  planned  nonconform¬ 
ance  at  PWC  San  Francisco.  Deductions  can  only  act  as  a 
motivator  where  payment  closely  follows  performance. 

Guidance  specifications,  when  developed,  should 
include  minimum  CPLs.  This  is  especially  true  where  the 
field  version  is  in  use,  as  inspection  levels  are  a  function 
of  the  CPL.  Activities  should  be  required  to  justify  lower 
CPLs.  This  study  indicates  that  a  CPL  of  90%  for  service 
calls  is  too  low. 

6 .  Computer  Assistance 

The  complexity  of  RSED  requires  computer  support 
over  a  range  of  operations.  This  support  is  necessary  to 
both  reduce  the  administrative  burden  and  minimize  the 
chance  of  error.  Programs  now  exist  which  are  close  to 
satisfying  this  need.  Full  service  programs  should  include: 
sample  generation  with  flexible  numbering,  sample  adjustment 
with  decision  routine,  payment  analysis,  and  sample  history 
if  appropriate  for  identifying  planned  samples. 

Computer  support  should  be  a  requirement  for  activi¬ 
ties  planning  on  implementing  a  RSED  program.  The  high 
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payback  of  random  sampling  would  justify  this  investment. 
Software  issued  by  NAVFAC  could  also  be  designed  to  help 
ensure  uniformity  in  the  application  of  RSED. 


B.  RSED  V3.0 

The  proposed  method  of  taking  extrapolated  deductions  is 
an  improvement  over  the  field  version  in  that  deductions  and 
inspection  requirements  are  not  a  function  of  the  CPL.  It 
provides  a  better  match  between  payment  and  performance  at 
lower  CPL  levels.  The  sample  sizes  required  by  RSED  V3 . 0 
are  approximately  equal  to  normal  sampling  at  a  CPL  of  95% 
and  reduced  sampling  at  97%.  Thus,  activities  with  a 
planned  CPL  of  90%  or  less  would  experience  a  significant 
increase  in  sample  requirements. 

Possible  disadvantages  associated  with  RSED  V3 . 0 
include: 

(1)  RSED  places  more  emphasis  on  proper  samples  due  to 
monthly  extrapolation. 

(2)  There  is  no  clear  measure  of  good  performance  which 
provides  financial  incentive. 

(3)  Where  contractor  performance  is  consistently  good, 
there  is  no  reduced  inspection  level  possible  because 
of  the  need  to  extrapolate. 

(4)  Influence  of  customer  complaints  is  diminished 
because  they  cannot  be  used  to  take  deductions. 

(5)  The  addition  of  a  payment  "adjustment"  is  one  more 
complication  added  to  the  payment  analysis.  Rounding 
procedures  must  be  clearly  addressed. 

The  importance  of  these  disadvantages  was  not  studied. 
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I  support  NAVFAC's  intentions  to  try  RSED  V3 . 0  because 
of  the  weakness  of  the  field  version  in  influencing 
performance  at  low  CPL  levels.  This  phenomenon  was  demon¬ 
strated  at  PWC  San  Francisco.  However,  results  from  the 
field  will  have  to  be  analyzed  before  a  definitive  statement 
can  be  made.  Interesting  comparisons  could  be  made  by 
shifting  PWC  San  Francisco  and  PWC  Great  Lakes  to  RSED  V3 . 0 
under  the  same  contractor. 


C.  RECOMMENDATIONS  FOR  FUTURE  STUDY 

1.  The  impact  of  RSED  V3 . 0  on  contract  price  and 
performance . 

2.  Relationship  between  inspector  aggressiveness  and  QA 
technique.  For  example,  if  an  inspector  knows  that 
every  defect  identified  will  have  ramifications  beyond 
its  own  importance,  will  he  be  less  likely  to  document 
borderline  cases? 

3.  A  cost/benefit  study  of  RSED  versus  planned  sampling. 

4.  A  legal  study  which  attempts  to  define  the  parameters 
within  which  NAVFAC  must  stay  when  designing  a  RSED 
program. 

5.  Identification  of  specific  contractors  which  serve 
both  the  Navy  and  Air  Force  under  their  respective 
RSED  programs.  Compare  payment  and  performance. 
Interview  the  contractor  for  his  perspective. 
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APPENDIX  A 


USING  MIL-STD-105D 

The  procedure  contained  in  this  appendix  is  taken  in 
part  from  OFPP  Pamphlet  No.  4,  "A  Guide  For  Writing  and 
Administering  Performance  Statements  of  Work  For  Service 
Contracts."  The  OFPP  approach  is  equivalent  to  Air  Force 
inspection  procedures  given  in  AFR  400-28.  A  sample  size 
and  corresponding  accept/reject  numbers  are  then  determined 
for  a  hypothetical  case,  along  with  the  associated 
producer/consumer  risk. 
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SAMPLING  PLAN 


A.  Deciding  on  the  Acceptable  Quality  Level  (AQL) .  The  AQL 
is  the  highest  number  of  defects  per  hundred,  highest 
percent  defective  or  highest  number  of  defects  that  can  be 
allowed  for  any  service  performance  indicator.  There  are 
only  a  limited  number  of  AQLs  listed  in  MIL-STD-105D  but,  in 
virtually  all  cases,  one  will  be  close  enough  to  control  the 
contractor's  level  of  service. 

(1)  The  first  step  in  designing  a  sampling  plan  under 
MIL-STD-105D  is  the  selection  of  a  realistic  AQL.  No 
service  can  be  perfectly  performed. 

(2)  Find  the  closest  AQL  from  Figure  1  and  use  it  to 
replace  the  original  AQL  on  the  Performance  Requirement 
Summary.  For  example,  the  AQL  for  taxi  service  might  have 
been  5  percent.  This  would  be  changed  to  4  percent  or  6.5 
percent  since  5  percent  does  not  appear  in  the  figure. 

B.  Determining  the  Lot  Size.  To  determine  the  sample  size, 
the  lot  size  must  be  known.  The  lot  is  how  often  the 
contractor  provides  the  service  in  a  period  of  time. 

(1)  To  determine  the  lot  size,  estimate  the  frequency 
of  the  service  to  be  sampled,  during  the  period  it  is  to  be 
sampled. 

(2)  In  the  case  of  workorders,  the  monthly  lot  size  can 
be  estimated  from  historical  information  on  file. 

C.  Determining  the  Sample  Size.  Use  Figure  2  to  identify 
an  appropriate  sample  size  for  a  given  lot  size. 

(1)  Use  the  normal  sample  size  column  unless  there  is  a 
limited  number  of  QAEs  or  unless  the  cost  of  an  inspection 
suggests  the  use  of  the  medium  or  small  sample  size  column. 

(2)  Use  the  medium  or  small  sample  size,  if  inspec¬ 
tions  for  a  particular  service  are  lengthy  or  hinder  the 
contractor's  ability  to  provide  service  to  customers. 

D.  Selecting  the  Rejection  Level.  Use  MIL-STD-105D  to 
identify  the  acceptance  and  rejection  level  for  the  sample 
size  (see  Figure  3)  .  To  use  the  figure,  begin  with  the 
known  values  for  the  AQL  and  the  sample  size. 

(1)  Find  the  selected  sample  size  (in  the  sample  size 
column)  and  read  across  that  line  to  the  column  for  the 
selected  AQL.  At  that  point  there  will  either  be  two 
numbers  or  an  arrow  pointing  up  or  down. 
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Allowable  Acceptable  Quality  Levels 

0.010  2 

1.0  2 

0.015  2 

1.5  2 

0.025  2 

2.5  2 

0.040  2 

4.0  2 

0.065  2 

6.5  2 

0.10  2 

10.  2 

0.15  2 

15.  2 

0.25  2 

25.  2 

0.40  2 

40.  2 

0.53  2 

65.  2 

Figure  1.  List  o-f  MIL-STD-105D  Acceptable  .Quality  Levels 
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Lot  Size 

Normal  Sample 
Size 

Medium  Sample 
Size 

Small  Sample 

Size 

2-8 

2 

2 

2 

9-15 

3 

2 

2 

16-25 

5 

3 

3 

26-50 

8 

5 

5 

51-90 

13 

5 

5 

91-150 

20 

8 

8 

151-280 

32 

13 

13 

281-500 

50 

20 

13 

501-1,200 

80 

32 

20 

1,201-3,200 

125 

50 

32 

3,201-10,000 

200 

80 

32 

10,001-35,000 

315 

125 

50 

35,001-150.000 

500 

200 

80 

150,001-500,000 

800 

315 

80 

500,000  and  over 

1250 

SOO 

125 

Figure  2.  Semple  Sizes 
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Figure  3.  MIL-STD-105D  Normal  Sampling  Table 


(2)  If  there  is  an  arrow,  follow  the  direction  of  the 
arrow  until  it  leads  to  a  pair  of  numbers.  Of  the  two 
numbers  at  the  intersection  or  at  the  end  of  the  arrow,  the 
number  on  the  left  (Ac  or  accept)  indicates  the  maximum 
number  of  defects  which  can  occur  in  a  sample  and  still 
permit  the  total  group  or  lot  to  be  judged  acceptable. 

(3)  When  there  is  no  accept  or  reject  number  for  a 
given  sample  size  and  AQL,  following  the  arrow  will  also 
cause  a  change  in  sample  size.  For  example,  with  an  AQL  of 
1.5  and  a  sample  size  of  20,  the  sample  size  would  become 
32. 


(4)  The  number  on  the  right  (Re  or  reject)  indicates 
the  minimum  number  of  defects  that  occur  in  a  sample  which 
causes  the  total  group  or  lot  to  be  judged  unacceptable. 
For  example,  suppose  the  sample  size  is  determined  to  be  32 
and  the  AQL  has  been  set  at  6.5  defects  per  hundred.  Find 
the  number  32  in  the  sample  size  column  and  read  across  that 
line  until  the  AQL  column  for  6.5  has  been  reached.  The  two 
numbers  at  that  intersection  are  5  and  6 . 
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EXAMPLE  CALCULATION 


Given  information: 

1.  Estimated  number  of  services  for  month:  1000 

2.  AQL:  6.5  percent 

3 .  Normal  inspection  desired 

Determined  from  the  tables: 

1.  Sample  size:  80 

2.  Accept  number:  10 

3.  Reject  number:  11 

It  should  be  noted  for  this  example  that  even  though  the 
activity  has  specified  6.5%  defective  as  satisfactory 
performance,  it  will  accept  10  defective  work  units  out  of  a 
sample  of  801.  This  equates  to  12.5%  of  the  sample  being 
defective. 

From  Figure  4  it  can  be  seen  that  if  the  actual  level  of 
defects  present  were  8.0%,  the  probability  of  acceptance 
would  be  95%.  This  equates  to  a  producer  (contractor)  risk 
of  5%.  At  the  same  time,  even  if  the  actual  level  is  as 
high  as  11%,  it  will  still  be  accepted  75%  of  the  time. 
Thus  consumer  (government)  risk  at  that  level  of  performance 
is  75%.  This  represents  a  large  bias  in  favor  of  the 
contractor. 
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CHART  C-G  •  OPERATING  CHARACTERISTIC  CURVES  FOR  NORMAL-TIGHTENED  SINGLE  SAMPLING  SCHEMES 


PERCC  -fT  OF  LOTS 
EXPC  ' E 0  TO  OE 
ACCE  r  t  C  D  (>••> 


5f 


Ho**',  f  <9iMca  an  tur*r»  mm  Act  Ou^irtf  Le-v-Ht  (A<X  t)  tor  wnwt  PyMtwtJ  twipicWy 

TABLE  C-G-TABULATED  VALUES  FOR  OPERATING  CHAnACl ElllST  1C  CURVES  FOR  NORMAL-TIGHTENED  SINGLE  SAMPLING  SCHEMES 


ACC  CP  1  A  III  £  QUALITY  iCVCIS 

N 

0  40 

1  *5 

|  40 

-IS  J 

1  '• 

0  10  j 

LiiJ 

’ll 

LilJ 

1  ••  1 

1  ••  ! 

1  " 

1  “  1 

L  ,o 

pocrc er*  O*  k-Clr»c| 

p‘.*—‘rcl»  or*  hum***  wmt 

99  0 

0  311 

0  410 

1  31 

2  M 

s  as 

9  71 

0031 

0  44  j 

1  34 

257 

5  55 

a  o« 

tea 

23  3 

39  3 

95  0 

0  ISC 

«  09 

2  42 

4  24 

a  oa 

12  7 

0  ISC 

t  oa 

7  42 

4  13 

7  71 

12  1 

ia  9 

:a  a 

44  0 

90  0 

0 

1  S3 

3  12 

5  17 

123 

14  2 

0  M 

1  M 

300 

5  07 

a  9/ 

13  9 

21  0 

313 

49  J 

n  o 

0  213 

2  40 

4  39 

•  to 

11.1 

1C  7 

0  VIC 

7  47 

4  37 

6  >4 

11.1 

15  4 

24  9 

35  a 

54  a 

500 

1  CO 

3  97 

932 

941 

14  0 

20  1 

1  91 

315 

9  43 

9  SO 

14  1 

TO  2 

295 

41  9 

C2  0 

25  0 

3  OS 

COO 

9  2C 

12  9 

17  a 

24  5 

3  11 

a  n 

950 

133 

ia  3 

25  3 

35  7 

49  4 

70  9 

«00 

5  00 

*52 

12  ) 

ICC 

27  0 

2  J  2 

5  U 

a  79 

13  3 

17  7 

73  3 

31  2 

43* 

57  4 

•02 

IS  2 

194 

25  0 

32  3 

C  71 

io  a 

n  i 

70  a 

2%  a 

35  2 

47  2 

12  9 

94  4 

I  0 

10  0 

14  C 

20  4 

250 

31  0 

it  5 

111  k 

19  c 

32  3 

jlfJ 

LLLi. 

43  a 

54  9 

73  5 

9*0 

Hol«:  Brtw«u<  UtUrA«w4iOi»  vwd  tor  porcon*  •cto'.wv*  co*H*rf4tMMV  tor  OrlecM  or*  waxtk. 


Figure  4.  Operating  Character i sti c  tor  Sample  Sice=32 
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APPENDIX  B 


MO-327  RANDOM  SAMPLING  PROCEDURES 


This  appendix  is  an  excerpt  from  the  current  NAVFAC  MO- 
327  entitled  "Service  Contracts:  Specifications  and 
Surveillance."  It  represents  the  NAVFAC  guidance  for  random 
sampling  prior  to  the  field  testing  of  extrapolated  deduc¬ 
tions.  The  MO-327  is  presented  here  to  illustrate  the 
limited  random  sampling  guidance  available  to  activities 
planning  to  begin  a  random  sampling  program.  Along  with  an 
explanation  of  random  sampling,  this  appendix  includes 
inspection  tables  and  a  procedure  to  generate  random 
numbers . 

When  NAVFAC  has  finalized  their  procedure  for  implement¬ 
ing  extrapolated  deductions  Navy-wide,  the  guidance  will 
appear  in  a  new  MO-327. 
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RANDOM  SAMPLING 


1.  GENERAL.  Random  sampling  is  a  surveillance  method  based 
on  statistical  theory.  The  key  element  of  random  sampling 
is  that  each  and  every  occurrence  of  work  has  an  equal 
chance  of  being  evaluated.  In  order  to  achieve  the  desired 
end  results,  knowledge  of  the  Contractor's  overall 
performance  based  on  evaluation  of  only  part  of  the  work, 
surveillance  by  random  sampling  must  be  applied  properly. 
Key  elements  of  random  sampling  are: 

a.  Sample  size  is  specified  for  a  given  population  to 
achieve  a  predetermined  level  of  statistical  accuracy. 

b.  The  sample  of  work  occurrences  selected  for 
evaluation  must  be  selected  by  a  random  process  in  which 
each  occurrence  has  equal  chance  for  selection. 

c.  Once  an  evaluation  schedule  has  been  established,  it 
must  be  followed  through  the  surveillance  period  (i.e., 
monthly  schedules) . 

d.  Surveillance  data  gathered  by  other  methods  (i.e., 
customer  complaints  and  unscheduled  inspections)  can  not  be 
combined  with  data  gathered  by  random  sampling. 

e.  Assessment  of  the  Contractor's  overall  performance, 
projected  from  the  observed  condition  of  the  sample,  will 
always  have  the  potential  to  be  in  error.  Statements  as  to 
overall  performance  should  be  stated  as  "The  Contractor's 
overall  defect  rate  is  in  excess  of  X  % . "  (Where  "X" 
equals  the  observed  defect  rate  minus  one  half  the  AQL — ex. 
ODR  (9.2%)  -  1/2  AQL  (10%)  =  4.2%. ) 

2.  MECHANICS  OF  RANDOM  SAMPLING.  Random  sampling  is  a 
structured  approach  based  on  statistics  to  contract 
surveillance.  As  such,  there  is  a  set  procedure  in  its 
application.  The  mechanics  of  applying  random  sampling  are 
as  follows. 

3.  POPULATION.  The  total  number  of  work  occurrences  for  a 
given  function  that  are  to  be  performed  during  the  surveil¬ 
lance  period  must  be  known  or  accurately  estimated.  The 
Inventory  of  Service  Worksheet  is  used  to  determine  popula¬ 
tion  size. 

a.  When  work  is  scheduled,  population  size  is  easy  to 
determine. 

Example:  Activity  X  has  80  dumpsters,  70  are 
emptied  weekly  and  10  are  emptied  twice  a  week.  The 
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population,  total  number  of  work  occurrences  per  month,  is 
360  (70x4)  +  (10  X  4  2) . 

b.  When  work  is  unscheduled,  population  size  must  be 
estimated. 

Example:  The  number  of  service  requests  for  the 

past  six  months  were: 

Jan  321 
Feb  301 
Mar  295 
Apr  337 
May  340 
Jun  320 


Total  1914 

The  average  number  of  service  calls  per  month  has  been  319 
(1914/6) .  This  would  be  the  expected  population  for  service 
calls  for  next  month  unless  there  is  some  known  reason  to 
expect  a  change. 

4.  SAMPLE  SIZE.  Sample  size  requirements  are  based  on  AQL, 
population  size,  and  level  of  surveillance.  Sample  size 
tables  are  used  to  determine  sample  size.  Tables  for  normal 
surveillance,  reduced  surveillance,  and  increased  surveil¬ 
lance  are  attached. 

a.  Select  the  table  with  the  desired  level  of  surveil¬ 
lance  (Tables  I,  II,  or  III). 

b.  Select  the  column  with  the  required  AQL  (.05,  .10, 

. 15 ,  . 20 ,  or  . 25) . 

c.  Select  the  row  that  is  closest  to  the  population 
size,  preferably  the  next  largest  entry. 

d.  The  number  indicated  by  the  row  and  column  selection 
is  the  sample  size  required  for  surveillance  in  one  surveil¬ 
lance  period. 

(1)  Daily  surveillance  requirements  will  be  deter¬ 
mined  by  dividing  the  required  period's  sample  size  by  the 
number  of  days  that  surveillance  is  to  be  conducted. 

(2)  Weekly  surveillance  requirements  will  be 
determined  by  dividing  the  required  sample  size  by  the 
number  of  weeks  in  a  period. 

(3)  When  computing  weekly  or  daily  sample  sizes, 

always  round  up  to  the  next  whole  number  (e.g.,  45  monthly 
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samples  required  and  2  0  work  days  per  month  results  in  a 
daily  sample  size  of  3  -  (45/20)  =  2.25  and  rounded  up 
results  to  3) . 

5.  Sample  Selection.  The  final  thing  to  be  decided  in 
sampling  is  how  the  sample  will  be  drawn.  The  objective  in 
the  method  is  to  insure  that  the  sample  is  random  (that  is, 
that  all  services  have  an  equal  chance  of  being  selected) . 
To  achieve  random  selection,  use  a  random  numbers  table  as 
explained  in  the  following  examples.  (A  random  numbers 
table,  Table  IV,  is  attached)  .  Most  items  will  fall  into 
one  of  these  examples. 

a.  Use  of  The  Random  Numbers  Table.  The  random  numbers 
in  Table  IV  are  arranged  in  groups  of  two. 

(1)  To  use  the  table,  begin  by  picking  at  random  a 
group  of  numbers  on  any  page  of  the  table.  This  is  usually 
done  by  closing  the  eyes  and  pointing  with  a  pencil  or 
finger  to  some  initial  group. 

(2)  To  identify  additional  random  numbers,  follow  a 
pattern.  Go  along  a  given  line  to  its  end  and  then  along 
the  next  line  to  its  end  and  so  on  through  the  table  until 
enough  numbers  have  been  selected  or  until  the  table  ends. 

(3)  If  the  table  ends  and  there  are  still  more 
numbers  to  select,  go  back  to  the  beginning  of  the  table  and 
continue  using  the  same  pattern.  Use  various  patterns 
alternately,  for  example,  use  lines  for  one  sample,  use 
columns  for  the  next  sample,  and  use  a  diagonal  pattern  for 
the  third  sample. 

b .  How  To  Use  the  Random  Numbers  Table  To  Identify  a 
Random  Sample  of  Consecutively  Numbered  Work  orders. 

Suppose  one  has  to  identify  a  random  sample  of  97  work 
orders  for  evaluation.  (Sample  size  is  based  on  a 
population  of  319  using  normal  surveillance.)  This  can  be 
done  at  the  beginning  of  the  month  (before  the  work  orders 
are  written)  or  at  the  end  of  the  month. 

(1)  If  there  are,  or  might  be,  319  consecutively 
numbered  work  orders  to  select  from,  then  one  begins  by 
listing  the  lowest  work  order  number  (known  or  projected) . 
This  cold  be  $001,  or  possibly  443,  or  any  other  sequen¬ 
tially  assigned  number.  List  the  highest  work  order  number 
(known  or  projected) ?  in  this  case,  it  could  be  #319  or  762. 
For  this  example,  use  work  orders  numbered  #443  to  762. 

(a)  Select  97  three  digit  numbers  from  Table  IV 
using  a  consistent  pattern. 
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(b)  If  random  numbers  selected  are  not  between 
443  and  762,  discard  the  number  outside  the  designated  range 
and  select  a  new  number. 

(2)  For  example,  using  the  initial  entry  on  Table 
IV  we  would  select  number  441.  This  number  is  too  low.  The 
next  number,  going  down  the  column,  is  343  again  too  low. 
The  third  number  selected  is  749.  This  number  falls  in  the 
range  of  work  order  number  (443-762)  subject  to  inspection. 
So  work  order  number  749  is  selected  to  be  inspected.  The 
next  work  order  number  selected  is  523. 

(3)  This  process  would  be  continued  until  three 
work  orders  are  selected. 

c.  How  To  Use  The  Random  Numbers  Table  To  Identify  A 
Random  Sample  From  A  Group  of  Items.  If  a  number  of  items 
need  to  be  sampled  that  are  not  consecutively  numbered,  the 
simplest  solution  is  to  list  the  identifiers,  for  all  the 
items  in  a  column,  on  a  piece  of  lined  paper. 

(1)  Next,  number  the  lines  consecutively,  beginning 
with  the  number  one.  Now  use  the  random  number  table  to 
draw  the  sample  from  the  line  numbers.  A  selected  line 
number  leads  to  the  identifier  located  on  that  line,  and 
that  identifier  tells  which  item  to  sample.  For  example,  if 
one  chooses  to  sample  a  set  of  work  orders  with  attached 
sales  slips,  one  is  not  going  to  have  to  have  a  set  of 
consecutively  numbered  work  orders  because  not  every  work 
order  has  a  sales  slip  attached. 

(2)  List  the  work  orders  with  sales  slips  in  a 
column,  number  each  line  in  the  column,  and  randomly  select 
enough  line  numbers  to  make  up  the  sample. 

d.  How  To  Use  The  Random  Numbers  Table  To  Identify  a 
Random  Samples  of  Days.  Suppose  one  wants  to  identify  four 
days  in  the  month  on  which  to  sample  something.  The  days  of 
the  month  can  be  numbered  01  to  31  (or  less,  as 
appropriate)  . 

(1)  It  is  best  to  use  a  starting  point  different 
from  the  one  used  in  the  previous  example.  For  the  purpose 
of  this  example,  it  is  being  used  again. 

(2)  One  can  move  down  the  column  from  number  to 
number  until  the  first  number  between  01  and  31  is  spotted. 
In  this  case,  it  is  22.  Thus  the  22nd  day  of  the  month  is 
selected  for  sampling. 

(3)  Continuing  in  this  fashion,  one  discovers  that 
11  is  the  next  number  selected.  This  number  is  disregarded. 
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Proceed  in  this  manner  until  the  four  days  for  sampling  have 
been  identified.  In  our  example,  the  4  days  selected  would 
be  22,  11,  10  and  24. 

(4)  If  it  is  not  desirable  to  sample  on  weekends, 
discard  those  days  selected  that  happen  to  fall  on  a  weekend 
and  continue  that  selection  until  the  proper  number  of  days 
has  been  selected. 

e .  How  To  use  the  Random  Numbers  Table  To  Identify  a 
Random  Sample  of  Times  of  Dav.  If  one  wants  to  select 
random  times  of  day  to  sample  a  service  such  as  taxi  or  bus 
service,  use  the  24  hour  clock. 

(1)  If  there  are  any  constraints  during  each  24- 
hour  period,  take  them  into  consideration.  For  example, 
suppose  that  base  bus  service  operates  between  0700  and 
2300.  Convert  these  times  to  minutes  (e.g.,  0700  =  0,  0410 
=  130,  1215  =  315,  etc.).  Again,  using  Table  IV  and 
selecting  three  digit  numbers  and  proceeding  across  the  line 
from  the  initial  number,  one  comes  to  441,  or  1421  hrs,  as 
the  first  random  time. 

(2)  The  next  random  number  is  34  3,  or  1243  hrs. 
The  number  is  good  and  so  one  schedules  an  observation  for 
1243  hrs. 

(3)  Proceed  in  this  manner  until  the  desired  number 
of  sample  times  have  been  identified. 

f .  How  To  Insure  Variety  in  the  Use  of  the  Random 
Numbers  Table.  The  use  of  variety  in  the  random  number 
table  ensures  that  detectable  patterns  do  not  occur. 

(1)  Success  in  using  the  tables  requires 
consistency  but  also  variety.  The  above  information  should 
ensure  that  the  tables  are  properly  used  and  that  the  sample 
is  randomly  drawn. 

g.  Other  Random  Numbers  Generating  Methods.  The  use  of 
a  hand  held  calculator  with  a  random  number  generating  capa¬ 
bility  is  an  alternative  to  the  use  of  random  numbers  table. 
Using  this  type  of  calculator  the  QAE  would  enter  the 
minimum  value  and  maximum  value  and  numbers  generated  would 
always  be  within  the  desired  range. 
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SAMPLE  SIZE  REQUIREMENTS  FOR  REDCC3D  SURVEILLANCE 
(SURVEILLANCE  LEVEL  I) 


POPULATION  AQL 


SHE 

.05 

.10 

.15 

.20 

.25 

50 

31 

21 

16 

11 

9 

75 

34 

25 

17 

13 

10 

100 

44 

27 

18 

13 

10 

125 

49 

28 

19 

14 

10 

150 

52 

30 

20 

14 

10 

175 

55 

30 

20 

14 

10 

200 

57 

31 

20 

14 

11 

225 

59 

32 

20 

14 

11 

250 

60 

32 

21 

14 

11 

275 

61 

32 

21 

15 

11 

300 

63 

33 

21 

15 

11 

325 

64 

33 

21 

15 

11 

350 

64 

33 

21 

15 

11 

375 

65 

33 

21 

15 

11 

400 

66 

34 

21 

15 

11 

425 

67 

34 

21 

15 

11 

450 

67 

34 

21 

15 

11 

475 

63 

34 

21 

15 

11 

500 

68 

34 

21 

15 

11 

550 

69 

34 

22 

15 

11 

600 

70 

34 

22 

15 

11 

650 

70 

35 

22 

15 

11 

700 

71 

3S 

22 

15 

11 

750 

71 

35 

22 

15 

11 

300 

72 

35 

22 

15 

11 

850 

72 

35 

22 

15 

11 

900 

73 

35 

22 

15 

11 

950 

73 

35 

22 

15 

11 

1000 

73 

35 

22 

15 

11 

1100 

74 

35 

22 

15 

u 

1200 

74 

35 

22 

15 

11 

uoo 

74 

36 

22 

15 

u 

1400 

75 

36 

22 

15 

11 

1500 

73 

36 

22 

15 

11 

1600 

75 

36 

22 

15 

u 

1700 

75 

36 

22 

15 

11 

1800 

76 

36 

22 

15 

11 

1900 

76 

36 

22 

15 

u 

2000 

76 

36 

22 

15 

11 

2500 

76 

36 

22 

15 

11 

3000 

77 

36 

22 

15 

11 

3500 

77 

36 

22 

15 

11 

4000 

77 

36 

22 

15 

11 

4500 

77 

36 

22 

15 

11 

5000 

78 

36 

22 

15 

11 

6000 

78 

36 

22 

15 

11 

7000 

78 

36 

22 

15 

11 

3000 

78 

36 

22 

15 

11 

9000 

78 

36 

22 

15 

11 

10000 

78 

36 

22 

15 

11 
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SAMPLE  SIZE  RECUIRIMETTrS  POR  NORMAL  SURVEILLANCE 
(SURVEILLANCE  LEVEL  II) 


POPULATION  AQL 


SIZE 

.05 

.10 

.15 

.20 

.25 

50 

43 

37 

32 

27 

23 

75 

60 

49 

40 

32 

27 

100 

76 

58 

46 

37 

29 

125 

89 

66 

51 

40 

31 

150 

101 

72 

54 

42 

32 

175 

111 

78 

57 

43 

33 

200 

121 

82 

60 

45 

34 

225 

129 

aV 

62 

46 

35 

250 

137 

89 

53 

47 

35 

275 

144 

92 

65 

48 

36 

300 

151 

95 

66 

48 

36 

325 

157 

97 

■67 

49 

37 

•  350 

162 

99 

68 

49 

37 

375 

167 

101 

69 

50 

37 

400 

172 

103 

70 

50 

37 

425 

176 

105 

71 

51 

38 

450 

181 

106 

71 

51 

38 

475 

184 

107 

72 

51 

38 

500 

iaa 

109 

72 

52 

38 

550 

195 

in 

73 

52 

38 

600 

201 

U3 

74 

52 

39 

650 

206 

114 

75 

52 

39 

700 

22  2 

115 

75 

53 

39 

750 

215 

117 

76 

53 

39 

aoo 

219 

ua 

76 

54 

39 

850 

222 

119 

77 

54 

39 

900 

226 

120 

77 

54 

39 

950 

229 

121 

78 

54 

39 

1000 

231 

122 

78 

54 

39 

1100 

236 

123 

78 

55 

40 

1200 

241 

124 

79 

55 

40 

1300 

244 

125 

79 

55 

40 

1400 

248 

126 

80 

55 

40 

1500 

251 

127 

80 

55 

40 

1600 

253 

128 

80 

55 

40 

1700 

256 

128 

80 

55 

40 

1800 

258 

129 

81 

56 

40 

1900 

260 

129 

81 

56 

40 

2000 

262 

130 

81 

56 

40 

2500 

269 

131 

82 

•  56 

40 

3000 

274 

132 

82 

56 

41 

3500 

277 

133 

82 

56 

41 

4000 

280 

134 

83 

57 

41 

4500 

282 

134 

83 

57 

41 

5000 

284 

135 

33 

57 

41 

6000 

287 

135 

83 

57 

41 

7000 

289 

136 

83 

57 

41 

3000 

290 

136 

84 

57 

41 

9000 

291 

136 

84 

57 

41 

2000 

292 

137 

84 

57 

41 
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SAMPLE  SIZE  REQUIREMENTS  TOR  INCREASED  SURVEILLANCE 
(SURVEILLANCE  LEVEL  III) 


POPULATION  AQL 


SIZE 

.05 

.10 

.15 

.20 

.25 

50 

47 

43 

39 

35 

31 

75 

67 

59 

52 

46 

39 

100 

86 

74 

63 

54 

45 

125 

104 

87 

72 

60 

50 

150 

121 

98 

80 

65 

53 

175 

136 

108 

96 

69 

56 

200 

151 

116 

92 

73 

58 

225 

164 

124 

97 

76 

60 

250 

177 

132 

101 

79 

62 

275 

189 

138 

105 

81 

63 

300 

201 

144 

108 

83 

64 

325 

211 

150 

111 

85 

65 

350 

222 

155 

114 

96 

66 

375 

231 

159 

116 

89 

67 

400 

241 

164 

119 

89 

68 

425 

249 

168 

121 

90 

69 

450 

258 

172 

123 

91 

69 

475 

266 

175 

124 

92 

70 

500 

273 

178 

126 

93 

70 

550 

288 

184 

129 

95 

71 

600 

301 

189 

132 

96 

72 

650 

313 

194 

134 

97 

72 

700 

324 

198 

136 

98 

73 

750 

334 

202 

138 

99 

74 

900 

344 

206 

139 

100 

74 

850 

352 

209 

141 

101 

75 

900 

361 

212 

142 

101 

75 

950 

368 

214 

143 

102 

75 

1000 

376 

217 

144 

103 

76 

noo 

389 

221 

146 

103 

76 

1200 

401 

225 

148 

104 

77 

1300 

411 

228 

149 

105 

77 

1400 

421 

231 

150 

106 

77 

1500 

429 

234 

151 

106 

77 

1600 

437 

236 

152 

107 

78 

1700 

444 

238 

153 

107 

78 

1800 

451 

240 

154 

107 

78 

1900 

457 

241 

155 

108 

78 

2000 

462 

243 

155 

108 

78 

2500 

485 

249 

158 

109 

79 

3000 

501 

253 

159 

uo 

80 

3500 

513 

256 

161 

111 

80 

4000 

523 

25  9 

161 

111 

90 

4500 

530 

260 

162 

111 

80 

5000 

537 

262 

163 

112 

90 

6000 

546 

264 

164 

112 

81 

7000 

554 

266 

164 

112 

81 

9000 

559 

267 

165 

113 

81 

9000 

563 

268 

16$ 

113 

31 

,0000 

567 

269 

165 

113 

81 

105 


SHORT  TABLE  OF  RANDOM  NUMBERS 


44 

19 

15 

32 

63 

55 

34 

39 

80 

62 

24 

33 

74 

97 

30 

30 

65 

07 

22 

14 

61 

60 

36 

38 

40 

03 

96 

40 

03 

47 

52 

33 

76 

44 

56 

15 

37 

59 

20 

40 

93 

17 

11 

02 

55 

57 

48 

84 

10 

33 

79 

26 

34 

54 

67 

59 

28 

25 

47 

39 

98 

50 

75 

20 

09 

13 

24 

43 

23 

72 

30 

64 

39 

91 

63 

13 

38 

27 

74 

62 

19 

67 

54 

13 

91 

03 

35 

60 

81 

16 

42 

57 

66 

76 

72 

91 

96  . 

36 

63 

06 

15 

03 

92 

70 

96 

70 

89 

80 

91 

08 

88 

53 

52 

13 

68 

85 

97 

74 

47 

53 

59 

54 

13 

09 

13 

80 

39 

18 

32 

69 

-  33 

46 

67 

43 

31 

09 

12 

60 

61 

75 

37 

19 

56 

90 

78 

10 

91 

11 

00 

63 

93 

23 

71 

58 

09 

78 

27 

55 

48 

32 

63 

89 

62 

13 

U 

71 

17 

23 

29 

89 

97 

47 

03 

13 

16 

94 

85 

82 

89 

07 

04 

S3 

10 

59 

75 

12 

95 

71 

43 

68 

97 

13 

86 

05 

39 

14 

35 

48 

59 

30 

60 

10 

41 

31 

05  . 

45 

35 

40 

54 

03 

71 

85 

17 

74 

66 

27 

80 

20 

32 

80 

98 

00 

13 

50 

78 

02 

73 

39 

67 

92 

65 

41 

45 

36 

72 

56 

73 

44 

26 

04 

28 

86 

85 

64 

94 

11 

69 

57 

40 

30 

44 

94 

71 

20 

03 

30 

79 

25 

89 

98 

55 

98 

22 

45 

58 

74 

82 

31 

14 

02 

50 

54 

73 

81 

91 

07 

49 

33 

72 

90 

10 

20 

11 

85 

01 

43 

65 

02 

34 

22 

46 

41 

84 

74 

42 

64 

64 

58 

22 

75 

84 

05 

72 

90 

44 

27 

23 

39 

94 

00 

80 

55 

04 

90 

51 

27 

61 

34 

77 

33 

29 

45 

00 

31 

67 

23 

11 

34 

79 

26 

30 

01 

84 

47 

45 

89 

71 

13 

33 

72 

08 

16 

60 

09 

21 

21 

18 

00 

75 

73 

73 

78 

19 

37 

24 

19 

90 

80 

87 

32 

36 

22 

67 

19 

20 

15 

33 

89 

74 

68 

48 

23 

65 

65 

20 

42 

23 

96 

34 

68 

02 

54 

37 

23 

27 

23 

46 

15 

36 

10 

78 

88 

84 

42 

32 

00 

92 

33 

69 

98 

96 

74 

97 

25 

14 

78 

21 

22 

63 

48 

46 

44 

01 

33 

33 

01 

58 

25 

37 

66 

14 

25 

49 

25 

94 

62 

82 

23 

00 

26 

36 

47 

05 

90 

14 

37 

48 

25 

29 

62 

03 

24 

64 

12 

19 

34 

03 

59 

23 

97 

57 

63 

73 

11 

80 

10 

39 

03 

56 

49 

92 

72 

63 

74 

58 

69 

03 

51 

03 

07 

71 

79 

32 

25 

59 

14 

72 

19 

17 

22 

25 

13 

85 

33 

35 

07 

86 

22 

45 

59 

98 

64 

30 

29 

89 

89 

80 

98 

84 

60 

93 

68 

16 

87 

17 

13 

08 

00 

50 

77 

13 

36 

57 

09 

62 

40 

69 

63 

77 

01 

89 

94 

96 

76 

27 

77 

94 

80 

19 

55 

56 

51 

36 

*  48 

92 

57 

51 

52 

33 

14 

82 

01 

28 

67 

51 

75 

96 

46 

21 

14 

39 

56 

81 

15 

35 

79 

26 

99 

78 

45 

36 

34 

45 

91 

82 

94 

93 

98 

01 

48 

17 

78 

.34 

54 

45 

04 

49 

82 

71 

57 

33 

28 

05 

77 

94 

65 

57 

70 

26 

25 

45 

49 

61 

22 

28 

44 

63 

95 

86 

75 

69 

56 

88 

34 

29 

64 

02 

57 

77 

47 

93 

72 

74 

91 

48 

46 

13 

34 

22 

07 

62 

17 

35 

34 

63 

27 

91 

70 

74 

13 

87 

44 

13 

50 

56 

48 

87 

31 

71 

33 

24 

47 

82 

74 

71 

11 

S4 

34 

10 

28 

61 

03 

72 

87 

04 

90 

42 

58 

19 

75 

19 

08 

92 

29 

20 

17 

98 

85 

63 

00 

98 

85 

40 

66 

77 

62 

58 

60 

74 

12  * 

01 

81 

65 

85 

27 

81 

78 

31 

63 


106 


51 

73 

03 

23 

73 

73 

96 

98 

97 

72 

47 

80 

41 

48 

44 

60 

83 

30 

33 

13 

19 

90 

63 

94 

02 

53 

* 

27 

88 

85 

29 

21 

36 

28 

10 

60 

78 

25 

59 

51 

10 

82 

92 

28 

67 

72 

47 

19 

52 

99 

00 

15 

31 

53 


40 

52 

17 

55 

OS 

36 

92 

34 

35 

25 

28 

56 

39 

07 

15 

79 

17 

37 

23 

24 

51 

25 

64 

53 

56 

97 

08 

38 

20 

42 

99 

06 

57 

51 

69 

75 

09 

31 

•93 

34 

34 

31 

75 

78 

46 

19 

09 

40 

29 

71 

37 

99 

99 

47 


29 

48 

30 

93 

45 

66 

29 

05 

86 

Cl 

35 

73 

57 

68 

36 

33 

91 

06 

9e 

;t 

43 

02 

42 

05 

32 

55 

02 

74 

39 

34 

24 

45 

79 

75 

33 

42 

00 

92 

33 

2^ 

13 

7« 

34 

89 

23 

39 

07 

17 

49 

13 

81 

^5 

32 

35 

70 

11 

17 

41 

64 

20 

30 

39 

87 

64 

37 

92 

35 

05 

43 

36 

98 

29 

97 

93 

07 

08 

30 

63 

21 

59 

69 

76 

02 

62 

31 

6  2 

47 

60 

92 

00 

04  ' 

94 

50 

05 

75 

82 

70 

80 

35 

11 

68 

25 

08 

95 

31 

79 

11 

79 

54 

05 

26 

37 

80 

39 

19 

06 

41 

02 

00 

53 

62 

59 

55 

05 

02 

16 

13 

17 

54 

43 

£6 

19 

29 

28 

76 

49 

74 

39 

50 

78 

26 

15 

41 

75 

64 

69 

61 

06 

38 

44 

04 

ca 

84 

80 

76 

51 

52 

41 

59 

01 

11 

05 

45 

11 

43 

40 

31 

84 

59 

43 

28 

10 

01 

65 

62 

07 

05 

59 

61 

31 

02 

65 

47 

47 

70 

39 

74 

22 

65 

97 

15 

70 

04 

89 

81 

73 

54 

84 

42 

95 

27 

52 

87 

47 

12 

52 

54 

62 

43 

38 

60 

36 

53 

56 

77 

06 

69 

03 

89 

91 

61 

04 

40 

33 

12 

06 

78 

91 

97 

88 

95 

20 

03 

64 

96 

60 

48 

01 

95 

44 

84 

69 

57 

92 

57 

11 

84 

44 

01 

33 

66 

53 

89 

81 

55 

87 

73 

81. 

58 

56 

42 

36 

25 

36 

49 

14 

34 

03 

52 

09 

20 

60 

11 

50 

46 

45 

88 

72 

50 

46 

11 

50 

46 

92 

45 

25 

48 

22 

23 

08 

32 

28 

87 

08 

74 

79 

91 

12 

43 

32 

03 

60 

19 

02 

70 

83 

72 

3  2 

20 

60 

86 

08 

64 

60 

44 

34 

34 

24 

S3 

77 

32 

92 

32 

44 

40 

47 

10 

33 

22 

52 

96 

55 

31 

99 

73 

23 

40 

07 

64 

54 

44 

66 

33 

97 

47 

58 

42 

44 

88 

09 

28 

58 

70 

15 

74 

43 

62 

69 

82 

30 

77 

23 

77 

22 

25 

94 

80 

62 

95 

48 

98 

23 

86 

38 

68 

36 

87 

81 

16 

77 

30 

19 

36 

50 

57 

77 

69 

60 

74 

22 

05 

77 

17 

77 

42 

59 

64 

99 

37 

03 

18 

03 

36 

69 

50 

59 

15 

79 

39 

42 

84 

18 

70 

39 

42 

48 

56 

84 

18 

70 

41 

74 

60 

88 

41 

20 

00 

15 

59 

60 

65 

65 

63 

78 

69 

24 

41 

63 

86 

10 

32 

00 

93 

35 

48 

15 

70 

11 

77 

83 

Cl 

91 

04 

02 

95 

63 

75 

74 

69 

65 

01 

34 

13- 

05 

’  57 

23 

06 

26 

23 

08 

66 

16 

81 

37 

78 

16 

05 

57 

12 

46 

22 

90 

37 

39 

23 

71 

15 

08 

82 

64 

87 

29 

01 

20 

05 

42 

67 

98 

41 

67 

44 

28 

71 

45 

08 

76 

05 

83 

03 

84 

32 

62 

83 

27 

48 

33 

84 

60 

46 

18 

41 

23 

74 

73 

51 

72 

90 

95 

32 

80 

64 

75 

91 

98 

09 

40 

64 

39 

46 

79 

86 

53 

77 

78 

06 

62 

37 

48 

82 

78 

45 

13 

23 

32 

01 

09 

46 

36 

43 

66 

35 

20 

60 

97 

48 

21 

41 

84 

22 

72 

77 

83 

67 

91 

44 

83 

43 

25 

56 

33 

23 

80 

27 

86 

50 

76 

93 

86 

35 

68 

45 

37 

93 
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APPENDIX  C 


SAMPLING  TABLES  USED  IN  THE  FIELD  TEST 

This  appendix  contains  tables  reproduced  from  those 
utilized  at  each  activity.  The  first  group  of  tables  were 
used  by  PWC  San  Francisco  at  the  CPL  =  90%  level.  Great 
Lakes  used  the  second  group  of  tables  at  the  CPL  =  97% 
level .  Both  groups  of  tables  were  generated  from  the  same 
formula.  The  tables  provide  graphic  illustration  of  the 
effect  that  the  CPL,  surveillance  level  and  population  have 
on  the  sample  size  and  CND. 
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C2TO/SS  I ABLE 


CRITICAL  NUMBER  OF  DEFECTIVES  (CSD)/ SAMPLE  SIZE  (SS) 


Reduced  Surveillance  CPagc  '1  of  2  ) 


> 

> 

> 

> 

WITS 

> 

CRITICAL  : 

P5RFORHA77CT 

LEVEL 

(CPL)   . 

> 

> 

OF  UORX 

> 

> 

> 

> 

> 

> 

> 

POPULATION  > 

95* 

> 

905: 

> 

85* 

> 

80* 

> 

75* 

> 

> 

SIZE 

> 

> 

> 

> 

> 

> 

> 

> 

OT D/SS 

> 

C37D/SS 

> 

CTD/SS 

> 

CTD/SS 

> 

CSD/SS 

> 

> 

50 

> 

3/36 

> 

5/27 

> 

6/21 

> 

6/17 

> 

6/1* 

> 

> 

75 

> 

4/47 

> 

6/33 

> 

6/25 

> 

7/19 

> 

7/26 

> 

> 

100 

> 

5/56 

> 

6/37 

> 

7/27 

> 

7/21 

> 

7/16 

> 

> 

125 

> 

6/62 

> 

7/40 

> 

7/29 

> 

7/22 

> 

7/17 

> 

> 

150 

> 

6/66 

> 

7/42 

> 

8/30 

> 

7/22 

> 

7/17 

*> 

> 

175 

> 

6/73 

> 

8/44 

> 

8/31 

> 

8/23 

> 

8/18 

> 

> 

200  . 

> 

7/77 

> 

6/46 

> 

8/31 

> 

8/23 

> 

8/18 

> 

> 

225 

> 

7/80 

> 

8/*7 

> 

8/32 

> 

8/24 

> 

8/18 

> 

> 

250 

> 

7/83 

> 

8/48 

> 

8/32 

> 

8/24- 

> 

8/18 

> 

> 

275 

> 

7/86 

> 

8/49 

> 

8/33 

> 

8/24 

> 

8/18 

> 

> 

300 

> 

8/88 

> 

8/49 

> 

8/33 

> 

8/24 

> 

8/19 

> 

> 

325 

> 

8/90 

> 

9/50 

> 

8/33 

> 

8/24 

> 

8/19 

> 

> 

350 

> 

8/92 

> 

9/51 

> 

9/34 

> 

8/24 

> 

8/19 

> 

> 

375 

> 

8/9* 

> 

9/51 

> 

9/34 

> 

8/25 

> 

8/19 

> 

> 

400 

> 

8/95 

> 

9/52 

> 

9/34 

> 

8/25 

> 

8/19 

> 

> 

425 

> 

8/97 

> 

9/52 

> 

9/34 

> 

8/25 

> 

8/19 

> 

> 

450 

> 

8/93 

> 

9/52 

> 

9/3* 

> 

8/25 

> 

8/19 

> 

> 

475 

> 

9/99 

> 

9/53 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

500 

> 

9/100 

> 

9/53 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

550 

>  ‘ 

9/102 

> 

9/53 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

600 

> 

9/103 

> 

9/54 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

650 

> 

9/105 

> 

9/54 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

700 

> 

9/106 

> 

9/55 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

750 

> 

9/107 

> 

9/55 

> 

9/35 

> 

8/25 

> 

8/19 

> 

> 

800 

>  . 

9/108 

> 

9/55 

> 

9/36 

> 

8/25 

> 

8/19 

> 

> 

850 

> 

9/109 

> 

9/55 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

900 

> 

9/110 

> 

10/56 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

950 

> 

9/110 

> 

10/56 

> 

9/36 

> 

9/26 

> 

8/19 

> 

1000 

> 

10/111 

> 

10/56 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1100 

> 

10/112 

> 

10/56 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1200 

> 

10/113 

> 

10/56 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1300 

> 

10/114 

> 

10/57 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1400 

> 

10/115 

> 

10/57 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1500 

> 

10/115 

> 

10/57 

> 

9/36- 

> 

9/26 

> 

8/19 

> 

> 

1600 

> 

10/116 

> 

10/57 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1700 

> 

10/116 

> 

10/57 

> 

9/36 

> 

9/26 

> 

8/19 

> 

> 

1800 

> 

10/ U  7 

> 

10/57 

> 

9/36 

> 

9/26 

> 

8/20 

> 

> 

1900 

> 

10/117 

> 

10/57 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

2000 

> 

10/118 

> 

10/57 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

2500 

> 

10/119 

> 

10/58 

> 

9/37 

> 

9/26 

> 

8/20 

> 
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CTD/SS  TABLE 


CRITICAL  NUMBER  OF  DEFECTIVES  <CHD)/SAMPL£  SIZE  (SS) 


Reduced  Surveillance  (Page  2  of  2  ) 


> 

> 

> 

> 

WITTS 

> 

CRITICAL  PERFORMANCE 

LEVEL 

CCPL) 

> 

> 

OF  WORK 

> 

> 

> 

> 

> 

> 

> 

POPULATION  > 

951 

> 

90* 

> 

851 

> 

801 

> 

751 

> 

> 

SIZE 

> 

> 

> 

> 

> 

> 

> 

> 

CTD/SS 

> 

CTD/SS 

> 

CTD/SS 

> 

CTD/SS 

> 

CTD/SS 

> 

> 

3000 

> 

10/120 

> 

10/58 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

3500 

> 

10/121 

> 

10/58 

> 

9/37 

y 

9/26 

> 

8/20 

> 

> 

4000 

> 

10/121 

> 

10/58 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

4500 

> 

10/122 

> 

10/58 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

5000 

> 

10/122 

> 

10/58 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

6000 

> 

10/122 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

7000 

> 

11/123 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

8000 

> 

11/123 

> 

10/59 

> 

9/37 

> 

9/26 

y 

8/20 

> 

> 

9000 

> 

11/123 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

10000 

> 

11/123 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

20000 

> 

11/124 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

30000 

> 

11/124 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

40000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

y 

> 

50000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

60000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

• 

70000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

80000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

90000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

100000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

250000 

y 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

> 

> 

500000 

> 

11/125 

> 

10/59 

> 

9/37 

> 

9/26 

> 

8/20 

>  • 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

y 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

>  * 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

y 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

ji 


no 


CND/SS  TABLE 


CRITICAL  NUH2ER  OF  DEFECTIVES  (CND) /SAMPLE  SIZE  CSS) 


Normal  Surveillance  (Page  1  of  2  ) 


> 

> 

> 

> 

CHITS 

> 

CRITICAL  PERFORMANCE 

LEVEL 

(CPU 

> 

> 

OF  WORK 

> 

> 

> 

> 

> 

> 

> 

POPULATION 

95X 

> 

901 

> 

85X 

> 

8  OX 

> 

75X 

> 

> 

SIZE 

> 

> 

> 

> 

> 

> 

> 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

> 

50 

> 

4/40 

> 

6/33 

> 

7/28 

> 

8/23 

> 

8/20 

> 

> 

75 

> 

5/55 

> 

7/42 

> 

9/34 

> 

9/27 

> 

9/23 

> 

> 

100 

> 

6/67 

> 

8/49 

> 

10/38 

> 

10/30 

> 

10/25 

> 

> 

125 

> 

7/78 

> 

9/55 

> 

10/41 

> 

11/32 

> 

11/26 

> 

> 

150 

> 

8/87 

> 

10/59 

> 

11/44 

> 

11/34 

> 

11/27 

> 

> 

175 

> 

8/95 

> 

11/63. 

> 

11/45 

> 

12/35 

> 

11/27 

> 

> 

200  . 

> 

9/101 

> 

11/66 

> 

12/47 

-  > 

12/36 

> 

11/28 

> 

> 

225 

> 

9/107 

> 

 .11/68/ 

> 

12/48 

> 

12/36 

> 

.  .11/28 

> 

> 

250 

> 

10/113 

> 

li/70 

> 

12/49 

> 

12/37 

> 

12/49 

> 

> 

275 

> 

10/118 

> 

12/72 

> 

12/50 

> 

12/37 

> 

12/29 

> 

> 

300 

> 

10/122 

> 

12/74 

> 

13/51 

> 

13/38 

> 

12/29 

> 

> 

325 

> 

11/126 

> 

13/75 

> 

13/52 

> 

13/38 

> 

12/30 

> 

> 

350 

> 

11/130 

> 

13/76 

>-  " 

13/52 

> 

13/39 

> 

12/30 

> 

> 

375 

> 

11/133 

> 

13/77 

> 

13/53 

> 

13/39 

> 

12/30 

> 

> 

400 

> 

12/136 

> 

13/78 

> 

13/53 

> 

13/39 

> 

12/30 

> 

> 

425 

> 

12/139 

> 

13/79 

> 

13/54 

> 

13/39 

> 

12/30 

> 

> 

450 

> 

12/141 

> 

13/80 

> 

13/54 

> 

13/39 

> 

12/30 

> 

> 

475 

> 

12/144 

> 

14/81 

> 

13/54 

> 

13/40 

> 

12/30 

> 

> 

500 

> 

12/146 

> 

14/82 

> 

14/55 

> 

13/40 

> 

12/30 

> 

> 

550 

> 

13/150 

> 

14/83 

> 

14/55 

> 

13/40 

> 

13/31 

> 

> 

600 

> 

13/153 

> 

.  14/84 

> 

14/56 

> 

13/40 

> 

13/31 

> 

> 

650 

> 

13/156 

> 

14/85 « 

> 

14/56 

> 

13/41 

> 

13/31 

> 

> 

700 

> 

13/159 

> 

14/86 

> 

14/56 

> 

13/41 

> 

13/31 

> 

> 

750 

> 

14/161 

> 

14/85 

> 

14/57 

> 

13/41 

> 

13/31 

> 

> 

800 

> 

14/164 

> 

15/87 

> 

14/57 

> 

13/41 

> 

13/31 

> 

> 

850 

> 

14/166 

> 

15/87 

> 

14/57 

> 

13/41 

> 

13/31 

> 

> 

900 

> 

14/167 

> 

15/88 

> 

14/57 

> 

13/41 

> 

13/31 

> 

> 

950 

> 

14/169 

> 

15/88 

> 

14/58 

> 

13/41 

> 

13/31 

> 

> 

*1000 

> 

14/172 

> 

15/89 

> 

14/58 

> 

13/41 

> 

13/31 

> 

> 

1100 

> 

15/173 

> 

15/89 

> 

14/58 

> 

14/42 

> 

13/32 

> 

> 

1200 

> 

15/176 

> 

15/90 

> 

14/58 

> 

14/42 

> 

13/32 

> 

> 

1300 

> 

15/178 

> 

15/91 

> 

15/59 

•> 

14/42 

> 

13/32 

> 

> 

1400 

> 

15/179 

> 

15/91 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

1500 

> 

15/181 

> 

15/91 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

1600 

> 

15/182 

> 

15/92 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

1700 

> 

15/183 

> 

15/92 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

1800 

> 

16/185 

> 

15/92 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

1900 

> 

16/186 

> 

15/93 

> 

15/59 

> 

14/42 

> 

13/32 

> 

> 

2000 

> 

16/186 

> 

15/93 

> 

15/60 

> 

14/42 

> 

13/32 

> 

> 

2500 

> 

15/190 

> 

16/94 

> 

15/60 

> 

14/43 

> 

13/32 

> 

m 


CND/SS  TABLE 


CRITICAL  NUMBER  OF  DEFECTIVES  (CND) /SAMPLE  SIZE  (SS) 


Normal  Surveillance  (Page  2  of  2  ) 


> 

> 

> 

> 

UNITS 

> 

CRITICAL  PERFORMANCE 

LEVEL 

(CPL) 

> 

OF  WORK 

> 

> 

> 

> 

> 

> 

> 

POPULATION 

> 

95* 

> 

90* 

> 

85* 

> 

80* 

> 

75* 

> 

> 

SIZE 

> 

> 

> 

> 

> 

> 

> 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

> 

3000 

> 

16/192 

> 

16/94 

> 

15/60 

> 

14/43 

> 

13/32 

> 

> 

3500 

> 

16/194 

> 

16/95 

> 

15/60 

> 

14/43 

> 

13/32 

> 

> 

4000 

> 

16/195 

> 

16/95 

> 

15/60 

> 

14/43 

> 

13/32 

> 

> 

4500 

> 

17/197 

> 

16/95 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

5000 

> 

17/198 

> 

16/96 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

6000 

> 

17/199 

> 

16/96 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

7000 

> 

17/200 

> 

16/96 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

8000 

> 

17/201 

> 

16/96 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

9000 

> 

17/201 

> 

16/96 

> 

15/61 

.  > 

14/43 

> 

13/32 

> 

> 

10000 

> 

17/202 

> 

16/96 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

20000 

> 

17/204 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

30000 

> 

17/204 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

40000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

50000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

60000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

70000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

80000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

90000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

100000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

250000 

> 

17/205 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

500000 

> 

17/206 

> 

16/97 

> 

15/61 

> 

14/43 

> 

13/32 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

*> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 
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CND/SS  TABLE 


CRITICAL  NUMBER  OF  DEFECTIVES  (CND) /SAMPLE  SIZE  (SS) 


> 
> 
> 
> 

>  SIZE  >>>>>> 


> 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

> 

50 

> 

4/45 

> 

7/40 

> 

9/36 

> 

11/32 

> 

11/28 

> 

> 

75 

> 

6/63 

> 

9/54 

> 

12/47 

> 

13/40 

> 

14/35 

> 

> 

100 

> 

7/80 

> 

11/66 

> 

24/55 

> 

15/46 

> 

16/39 

> 

> 

125 

> 

8/95 

> 

13/76 

> 

15/62 

> 

17/51 

> 

17/43 

> 

;iso 

> 

9/110 

> 

14/85  * 

> 

17/68 

> 

18/55 

> 

18/43 

> 

> 

175 

> 

10/123 

> 

15/92 

> 

18/72 

> 

19/56 

> 

19/47 

> 

> 

200 

> 

11/135 

> 

16/99 

> 

19/76 

> 

20/60 

> 

20/49 

> 

> 

225 

> 

12/145 

> 

17/104 

> 

19/79 

> 

20/63 

> 

20/50 

> 

> 

250 

> 

13/156 

> 

18/110 

> 

20/82 

> 

21/64 

> 

21/52 

> 

> 

275 

> 

14/165 

> 

19/114 

> 

21/85 

> 

21/65 

> 

21/53 

> 

> 

300 

> 

15/174 

> 

20/118 

> 

21/87 

>  . 

22/67 

> 

21/53 

> 

> 

325 

> 

15/182 

> 

20/122 

> 

22/89 

> 

22/68 

> 

22/54 

> 

> 

350 

> 

16/189 

> 

21/125 

> 

22/91 

> 

22/69 

> 

22/55 

> 

.  375 

> 

16/196 

> 

21/128 

> 

23/92 

> 

23/70 

> 

22/55 

> 

400 

> 

17/203 

> 

22/131 

> 

23/94 

> 

23/70 

> 

22/56 

> 

> 

425 

> 

18/209 

> 

22/134 

> 

23/95 

> 

23/72 

> 

22/56 

> 

> 

450 

> 

18/215 

> 

22/136 

> 

24/96 

.  > 

26/73 

> 

23/57 

> 

> 

475 

> 

18/220 

> 

23/133 

> 

24/98 

> 

24/73 

> 

23/57 

> 

> 

500 

> 

19/226 

> 

23/140 

> 

24/99 

> 

24/74 

> 

23/58 

> 

> 

550 

> 

20/235 

> 

24/144 

> 

24/100 

> 

24/75 

> 

23/58 

> 

> 

600 

> 

20/244 

> 

24/147 

> 

25/102 

> 

25/76 

> 

24/59 

> 

> 

650 

> 

21/252 

> 

25/150 

> 

25/103 

> 

25/76 

> 

24/59 

> 

> 

700 

> 

22/259 

> 

25/152 

> 

25/104 

> 

25/77 

> 

24/59 

> 

> 

750 

> 

22/266 

> 

25/155 

> 

26/105 

> 

25/78 

> 

24/60 

> 

> 

800 

> 

23/272 

> 

26/157 

> 

26/106 

> 

25/78 

> 

24/60 

> 

> 

850 

> 

23/277 

> 

26/159 

> 

26/107 

> 

25/79 

> 

24/60 

> 

> 

900 

> 

23/282 

> 

26/159 

> 

26/107 

> 

25/79 

> 

24/60 

> 

> 

950 

> 

24/287 

> 

27/162 

> 

27/109 

> 

26/79 

> 

24/61 

> 

> 

1000 

> 

24/292 

> 

27/163 

> 

27/109 

> 

26/80 

> 

24/61  . 

> 

> 

1100 

> 

25/299 

> 

27/166 

> 

27/110 

> 

26/80 

> 

24/61 

> 

> 

1200 

> 

25/306 

> 

28/168 

> 

27/111 

> 

26/81 

> 

25/62 

> 

> 

1300 

> 

26/313 

> 

28/170 

> 

27/112 

> 

26/81 

> 

25/62 

> 

> 

1400 

> 

26/318 

> 

28/171 

> 

28/113 

> 

26/82 

> 

25/62 

> 

> 

1500 

> 

27/323 

> 

28/173 

> 

28/113 

> 

26/82 

> 

25/62 

> 

> 

1600 

> 

27/327 

> 

29/174 

> 

28/114 

> 

26/82 

> 

25/62 

> 

> 

1700 

> 

27/331 

> 

29/175 

> 

28/114 

> 

26/82 

> 

25/63 

> 

> 

iaoo 

> 

28/335 

> 

29/176 

> 

28/115 

> 

27/83 

> 

25/63 

> 

> 

1900 

> 

28/335 

> 

29/177 

> 

28/115 

> 

27/83 

> 

*  25/63 

> 

2000 

> 

28/341 

> 

29/178 

> 

28/116 

> 

27/83 

> 

25/63 

> 

2500 

> 

29/353 

> 

30/181 

> 

29/117 

> 

27/84 

> 

25/63 

> 

Tightened  Surveillance  (Page  1  of  2  )    

>  > 

>  UNITS  > _ CRITICAL  PERFORMANCE  LEVEL  CCPLj _ 

>  OF  WORK  >  >  >  >  > 

>  POPULATION  >  95* _ >  90* >  '  85* _ > _ 80*  _  _>_  _ 75* 
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CND/SS  TABLE 


CRITICAL  NUMBER  OF  DEFECTIVES  (CND)ZSAMPLE  SIZE  (SS) 


Tightened  Surveillance  (Page  2  of  2  ) 


> 

> 

> 

> 

units 

> 

CRITICAL  PERFORMANCE 

LEVEL 

(CPL) 

> 

> 

OF  WORK 

> 

> 

> 

> 

> 

> 

> 

POPULATION 

> 

95* 

> 

9  OX 

> 

85X 

> 

80X 

> 

75X 

> 

> 

SIZE 

> 

> 

> 

> 

> 

> 

> 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

CND/SS 

> 

> 

3000 

> 

30/362 

> 

30/183 

> 

29/118 

> 

27/84 

> 

25/64 

> 

> 

3500 

> 

30/368 

> 

30/185 

> 

29/119 

> 

27/85 

> 

25/64 

> 

> 

4000 

> 

31/373 

> 

30/186 

> 

29/119 

> 

27/85 

> 

26/64 

> 

> 

4500 

> 

31/377 

> 

31/187 

> 

29/119 

> 

27/85 

> 

26/64 

> 

> 

5000 

> 

31/380 

> 

31/188 

> 

29/120 

> 

27/85 

> 

26/64 

> 

> 

6000 

> 

31/385 

> 

31/189 

> 

29/120 

> 

27/85 

> 

26/64 

> 

> 

7000 

> 

31/389 

> 

31/190 

> 

29/121 

> 

28/86 

> 

26/64 

> 

> 

8000 

> 

31/391 

> 

31/190 

> 

29/121  * 

> 

28/86 

> 

26/64 

> 

> 

9000 

> 

31/394 

> 

31/191 

> 

30/121 

> 

28/86 

> 

26/65 

> 

> 

10000 

> 

31/395 

> 

31/191 

> 

30/121 

> 

28/86 

> 

26/65 

> 

> 

20000 

> 

31/403 

> 

31/193 

> 

30/122 

> 

28/86 

> 

26/65 

> 

> 

30000 

> 

31/406 

> 

31/194 

> 

30/122 

> 

28/86 

> 

26/65 

> 

> 

40000 

> 

31/407 

> 

31/194 

> 

30/122 

> 

28/86 

> 

26/65 

> 

> 

50000 

> 

31/408 

> 

31/194 

> 

30/122 

> 

28/86 

> 

26/65 

> 

> 

60000 

> 

31/409 

> 

31/194 

> 

30/122 

> 

28/87 

> 

26/6  5 

> 

> 

70000 

> 

31/409 

> 

31/194 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

80000 

> 

31/409 

> 

31/194 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

90000 

> 

31/410 

> 

31/195 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

100000 

> 

31/410 

> 

31/195 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

250000 

> 

31/411 

> 

31/195 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

500000 

> 

31/411  . 

> 

31/195 

> 

30/123 

> 

28/87 

> 

26/65 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

’  > 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

>' 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 

> 
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CND/SS  TABLE 


REDUCED  SURVEILLANCE 


N 

997. 

987. 

97%  96%  93% 

M 

1  /  44 

2/43 

2/40  3  / 

30  3  / 

34 

79 

2/47 

2/41 

3/99  4  / 

91  4  / 

47 

100 

2  /  *7 

3/74 

4/40  4 

/  41  9  / 

94 

129 

2  /  109 

3/40 

4/74  9 

/  70  4 

/  42 

190 

2  /  122 

4/102 

9  /  ao 

4/77  4 

/  40 

1  79 

3  /  13» 

4  /  113 

9/44 

*  /  EX  4 

/  73 

200 

3/193 

9/123 

4  /  103 

4/00 

7/77 

229 

3  /  147 

9/132 

4  /  104 

7/43 

7  /  ao 

290 

3  /  101 

9  /  Ml 

4  /  119 

7/47 

7/03 

779 

4  ✓  143 

9  /  140 

4  /  120 

7/100 

7  /  04 

TOO 

4  /  209 

4  /  199 

7  /  124 

7/103 

a  /  ao 

329* 

4/217 

4/142 

7/124 

7/104 

a  /  ao 

390 

4  /  22% 

4/140 

7/132 

0  /  104 

•  /  42 

3T9 

4  /  230 

4/173 

7/134 

•  /  111 

a  /  44 

400 

9  /  240 

4  /  170 

7/134 

•  /  113 

a  /  49 

429 

9  /  297 

4/103 

7/1  <2 

•  /  119 

a  /  47 

490 

9  /  244 

7/100 

•  /  M4 

a  /*  1 1  7 

a  /  to 

479 

9/279 

7/142 

■  /  147 

a  /  na 

4/44 

900 

9/203 

7  /  144 

■  /  144 

a  /  120 

4/100 

990 

9/240 

7/203 

■  /  193 

4/123 

4  /  102 

400 

4/312 

7  /  210 

•  /  197 

4  /  129  . 

4  /  103 

490 

4/329 

■  /  219 

0/140 

4/127 

4  /  109 

TOO 

4/337 

0/221 

0/143 

4/124 

4/104 

790 

4/340 

■  /  229 

4  /  144 

4/190 

4  /  107 

OOO 

4  /  394 

0/230 

4/140 

4/132 

4/100 

■90 

7  /  344 

0/234 

4  /  170 

4  /  133 

4/104 

400 

7  /  370 

0  /  237 

4/172 

4/134 

4  /  no 

490 

7  /  9M 

0/941 

4  /  174 

4/139 

4  /  no 

lOOO 

7  /  344 

0/244 

4/179 

4/134 

10  /  111 

1100 

7  /  404 

4  /  244 

4/170 

4  /  13i 

10  /  112 

1200 

7  /  422 

4/294 

4  /  101 

10  /  134 

10  /  113 

1300 

•  /  434 

4/290 

4  /  1  S3 

10  /  141 

10  /  114 

1400 

•  /  444 

4/24 2 

10  /  109 

10  /  142 

10  /  119 

1900 

■  /  494 

4  /  249 

10  /  104 

10  /  143 

10  /  119 

1400 

■  /  443 

4  /  240 

10  /  IN 

10  /  144 

10  /  It* 

1700 

■  /  471 

4/271 

10  /  104 

10  /  144 

10  /  114 

l«O0 

■  /  470 

4  /  273 

10  /  140 

10  /  149 

10  /  117 

1400 

■  /  409 

4/279 

10  /  141 

10  /  144 

10  /  117 

2000 

4  /  441 

10/277 

10  /  142 

10  /  144 

io  /  no 

2900 

4  /  914 

10  /  209 

10  /  144 

10  /  140 

10  /  114 

3000 

4  /  939 

10  /  74| 

10  /  1 40 

10  /  190 

10  /  120 

3900 

4  /  >44 

10/249 

10  /  200 

10  /  191 

10  /  121 

4000 

10/940 

10/240  10  /  202 

10  /  192  10  /  121 

115 


CND/SS  TABLE 


NORMAL  SURVEILLANCE 

N  997.  987.  977.  967.  93X 

90  |/4#  2  /  4*  3  /  44  3/  43  4/40 

79  3/70  3/44  4/43  4/90  3/99 


too 

2  / 

71 

3  /  #4  4 

/  70  9 

/  73  4  / 

47 

139 

3  / 

112 

4  /  101 

3/72 

4/84  7 

/  78 

1  30 

3  / 

132 

4  /  117 

4/103 

7/73  8 

/  #7 

173 

3  / 

130 

3/132 

4  /  117 

7  /  103 

8/79 

300 

3  / 

147 

3  /  149 

7/127 

8  /  113 

7  /  101 

239 

4  / 

1  #4 

4  /  19# 

7  /  137 

8  /  121 

7  /  107 

390 

4  / 

203 

4/170 

■  /  144 

7/127 

10  /  113 

373 

4  / 

217 

4  /  101 

3/194 

7/134 

10  /  118 

4  / 

234 

7/173 

3/142 

10  /  137 

10  /  123 

333 

3  / 

347 

7  /  303 

7  /  147 

10  /  144 

11  /  124 

390 

3  / 

344 

7  /  211 

7  /  173 

10  /  147 

11  /  130 

373 

3  / 

27# 

■  /  220 

7  /  181 

10  /  193 

11  /  133 

400 

3  / 

271 

•  /  22# 

10  /  1 #7 

11  /  497 

12  /  134 

433 

9  / 

304 

•  /  234 

10  /  173 

11  /  141 

12  /  137 

490 

4  / 

317 

•  /  243 

10  /  177 

11  /  149 

12  /  141 

479 

4  / 

327 

7/391 

10/202 

11  /  14# 

12  /  144 

900 

4  / 

341 

7  /  237 

1L  /  204 

12  /  171 

12  /  144 

990 

4  / 

343 

7/270 

11  /  214 

12  /  174 

13  /  190 

400 

7  / 

303 

10  /  7S2 

11  /  221 

12  /  181 

13  /  193 

430 

7  / 

403 

10  /  273 

12  /  227 

13  /  184 

13  /  134 

700 

7  / 

423 

10  /  302 

12  /  403 

13  /  I #7 

13  /  197 

790 

•  / 

441 

11  /  311 

12/237 

13  /  173 

14  /  141 

400 

■  / 

43# 

11  /  317 

12  /  243 

13  /  174 

14  /  144 

•90 

•  / 

474 

11/327 

13  /  248 

13  /  177 

14  /  144 

400 

7  / 

407 

11/334 

13/292 

14  /  203 

14  /  147 

790 

7  ✓ 

304 

12  /  340 

13/234 

14  /  204 

14  /  147 

10 oo 

7  /  317 

12  /  347 

13  /  297 

14  /  304 

14  /  171 

noo 

7  i 

'  343 

12  /  33# 

14  /  344 

14  /  210 

13  /  173 

1300 

10 

/  944 

13/34# 

14  /  271 

14  /  214 

19  /  174 

1300 

10 

/  907 

13  /  377 

14  /  274 

19  /  317 

13  /  17# 

1400 

10 

/  407 

13  /  3*9 

14  /  340 

13  /  317 

13  /  177 

1900 

1 1 

/  439 

13  /  372 

14  /  244 

19  /  221 

19  /  181 

1400 

11 

/  442 

14  /  370 

19  /  2#7 

13  /  223 

19  f  102 

1700 

11 

/  437 

14  /  404 

19  /  270 

13/233 

13  /  103 

1400 

13 

/  473 

14  /  410 

13  /  273 

13/227 

14  /  109 

1700 

12 

/  409 

14  /  419 

19/24* 

19  /  237 

14  /  104 

3000 

12 

/  47# 

14  /  4|7 

13/273 

13  /  230 

14  /  104 

3900 

13 

/  790 

13  /  430 

14  /  307 

14  /  239 

14  /  170 

3000 

13 

/  770 

13  /  491 

14  /  313 

14/237 

14  /  172 

3900 

14 

/  #20 

14  /  441 

14  /  318 

14  /  242 

14  /  174 

4000 

14 

/  44  9 

14  /  44# 

14/323 

14  /  244 

14  /  174 
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CND/SS  TABLE 


TIGHT  SURVEILLANCE 


N  99%  98%  97%  96%  93% 


30 

1  /  44 

2  /  4t  3 

/  47  3  / 

44  4  /  43 

79 

too 

2/72 

2/44 

3/70  4 

4/41  3 

/  40  3  / 

/  M  4  / 

44  0/03 

•4  7/00 

its 

2  /  lit 

4  /  112 

4  /  104 

7/101  0 

/  44 

190 

3  /  140 

3  /  131 

7  /  124 

0  /  110  4 

/  110 

179 

3/142 

3/130 

7  /  140 

4  /  131  10  /  123 

300 

3  /  1*3 

4  /  140 

•  /  134 

10  /  144 

11  /  133 

339 

4  /  204 

7  /  104 

4  /  170 

11 ./  137 

12  /  143 

290 

4  /  224 

7  /  202 

10  /  104 

12  /  104 

13  /  130 

379 

4  /  244 

•  /  210 

10  /  147 

12  /  100 

14  /  103 

300 

•  /  203 

0/330 

11  /  tio 

13  /  140 

10  /  174 

xn 

0/303 

0  /  >44 

11  /  22 

14  /  900 

1  §  /  103 

300 

3/301 

4  /  Ml 

Lf  /  ro 

14  /  0 04 

lO  /  104 

379 

4  /  314 

10  /  3T7 

12  /  244 

13  /  210 

10  /  140 

400 

4/337 

10  /  241 

13  /  233 

13/220 

17  /  203 

423 

4  /  333 

10  /  303 

13  /  244 

lO  /  234 

10  /  204 

430 

7/372 

11  /  314 

14  /  274 

14  /  241 

10  /  213 

473 

7  /  304 

11/32* 

14  /  2*3 

17  /  240 

10  /  220 

500  . 

7  /  403 

12/340 

IS  /  2*2 

17  /  233 

14  /  234 

330 

•  /  430 

12/342 

14  /  300 

10  /  247 

20  /  233 

*oo 

•  /  444 

13  /  303 

14/333 

14  /  274 

20/244 

430 

4  /  444 

14  /  403 

17  /  337 

14  /  204 

21  /  232 

700 

4  /  320 

14  /  422 

10  /  330 

20  /  240 

33  /  234 

730 

10  /  334 

13  /  434 

lO  /  342 

21  /  307 

33  /  200 

•OO 

10/303 

13  /  434 

14/373 

21  /  313 

23/272 

•30 

10  /  404 

14  /  472 

14  /  304 

33  /  323 

23  /  277 

400 

11  /  434 

14  /  *07 

20/344 

33/300 

23  /  as 

430 

11  /  430 

17  /  301 

30  /  403 

33/300 

24  /  207 

lOOO 

12  /  4*2 

17  /  313 

21  /  412 

23/342 

24  /  2*2 

1100 

12  /  727 

!•  /  3*0 

31  /  420 

34  /  333 

23  /  240 

1200 

13  /  744 

14/342 

33  /  442 

24  /  343 

23  /  304 

1300 

14  /  004 

20/304 

23  /  433 

23  /  371 

24  /  313 

1400 

14  /  447 

20  /  404 

23  /  447 

23/374 

24  /  310 

1300 

19  /  W 

21/422 

*24  /  477 

20  /  300 

27/333 

1400 

14  /  414 

21  /  43B 

24  /  407 

24  /  342 

27  /  327 

1700 

14  /  440 

22  /  433 

23  /  *44 

20  /  340 

27/331 

lOOO 

17  /  474 

22/440 

23  /  304 

27/403 

20/333 

1400 

17  /  1007 

23/401 

24  /  312 

27  /  400 

20  /  330 

2000 

17  /  1033 

23  /  443 

24  /  314 

27  /  413 

30  /  341 

2300 

14  /  1134 

23  /  743 

27  /  347 

24  /  430 

24  /  333 

3000 

31  /  1230 

24  /  704 

20  /  340 

24  /  443 

30/342 

3300 

22  /  1330 

27  /  014 

24  /  304 

30  /  433 

30/340 

4000 

23  /  1344 

30  /  B34 

30/944 

30  /  400 

31/373 
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APPENDIX  D 


RSED  V3.0  TABLES  AND  PAYMENT  ANALYSIS 


This  appendix  contains  inspection  tables  and  deduction 
adjustments  (deltas)  for  RSED  V3 . 0 .  That  is  followed  by  a 
sample  payment  analysis.  The  material  within  this  appendix 
has  been  reproduced  from  preliminary  NAVFAC  guidance  on  the 
proposed  method  [Ref.  19]  .  Interesting  features  are  the 
enhanced  inspection  requirements  and  narrowly  defined 
population  ranges.  There  is  no  CND  because  extrapolation 
takes  place  in  the  event  of  any  defect. 
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RSED  V3.0  -  Table  o-f  Sample  Sizes 
For  Normal  Sampling  Levels 

07-07-1986 

Sample  sizes  are  for  the  indicated  monthly  population: 


1 1  a t i on 

range  - 

Sampl.e  Size 

Popu 1  at i on 

range  - 

Samp  1 e  Size 

33 

_ 

34.  .  , 

.  30 

35  - 

36.  .  . 

37 

- 

37.  .  . 

.  32 

38  - 

38.  .  . 

. .  33 

39 

- 

40.  .  . 

41  - 

41  .  .  . 

. .  35 

42 

- 

43.  .  . 

. .  36 

44  - 

44.  .  . 

45 

- 

46.  .  . 

.  38 

47  - 

47.  .  . 

48 

- 

49.  .  . 

. .  40 

50  - 

50.  .  . 

. .  4  1 

51 

- 

52.  .  . 

53  - 

53.  .  . 

. .  43 

54 

- 

55.  .  . 

. .  44 

56  - 

56.  .  . 

.  45 

57 

- 

58.  .  . 

.  46 

59  - 

60.  .  . 

. .  47 

61 

- 

61  .  .  . 

62  - 

63.  .  . 

.  49 

64 

- 

65.  .  . 

66  - 

66 .  .  . 

. .  51 

67 

- 

68.  .  . 

. .  52 

69  - 

70.  .  . 

71 

- 

71  .  .  . 

. .  54' 

72  - 

73.  .  . 

74 

- 

75.  .  . 

.  56 

76  - 

77.  .  . 

.  57 

78 

- 

79.  .  . 

. .  58 

80  - 

81  .  .  . 

.  59 

82 

- 

83.  .  . 

84  - 

84.  .  . 

85 

- 

86.  .  . 

. .  62 

87  - 

88.  .  . 

.  63 

89 

- 

90.  .  . 

91  - 

92.  .  . 

93 

- 

95.  .  . 

. .  66 

96  - 

97.  .  . 

98 

- 

99.  .  . 

.  68 

100  - 

101 .. . 

102 

- 

103. . . 

. .  70 

104  - 

105. . . 

.  71 

106 

- 

107. . . 

.  72 

108  - 

110... 

.  73 

1  1  1 

- 

112... 

113  - 

114... 

115 

- 

117... 

.  76 

118  - 

119... 

120 

- 

122. . . 

. .  78 

123  - 

124. . . 

125 

- 

127.  .  . 

128  - 

129. . . 

.  81 

130 

- 

132. . . 

133  - 

134. . . 

.  83 

135 

- 

137. . . 

138  - 

140. . . 

141 

- 

142. . . 

.  86 

143  - 

145. . . 

146 

- 

148. . . 

.  88 

149  - 

151 .. . 

152 

- 

154.  .  . 

.  90 

155  - 

157. . . 

.  91 

158 

- 

160. . . 

.  92 

161  - 

163. . . 

.  93 

164 

- 

166. . . 

.  94 

167  - 

169. . . 

170 

- 

172. . . 

173  - 

176. . . 

.  97 

177 

- 

179. . . 

180  - 

182. . . 

.  99 

183 

- 

186. . . 

.  100 

187  - 

189. . . 

.  101 

190 

- 

193. . . 

.  102 

194  - 

196. . . 

.  103 

197 

- 

200. . . 

.  104 

201  - 

204. . . 

205 

_ 

208. . . 

.  106 

209  - 

211... 

.  107 

212 

- 

215. . . 

216  - 

219. . . 

.  109 

220 

- 

224. . . 

.  1  10 

225  - 

228. . . 

.  Ill 

229 

- 

232. . . 

.  112 

233  - 

236. . . 

.  113 

237 

- 

241 .. . 

.  114 

242  - 

245. . . 

.  115 

246 

- 

250. . . 

251  - 

255. . . 

.  117 

256 

- 

259. . . 

260  - 

264. . . 

265 

- 

269. . . 

.  120 

270  - 

274. . . 

275 

- 

‘  280... 

.  122 

281  - 

285. . . 
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RSED  V3.0  -  Table  of  Sample  Sizes 
For  Normal  Sampling  Levels 


Sample  sizes  are  -for  the  indicat 


Population  range  -  Sample  Size 

206  -  290 .  124 

297  -  302 .  126 

309  -  313 .  120 

321  -  326 .  130 

333  -  339 .  132 

346  -  352 .  134 

360  -  366 .  136 

375  -  381 . .  130 

390  -  397 .  140 

406  -  414 .  142 

423  -  431 .  144 

441  -  450 .  146 

460  -  469 .  140* 

480  -  490 .  150 

502  -  512 .  152 

524  -  535 .  154 

54?  -  560 .  156 

575  -  507 .  150 

602  -  616 .  160 

632  -  647 .  162 

664  -  600 .  164 

698  -  716 .  166 

736  -  754 .  160 

776  -  796 .  170 

020  -  042 .  172 

860  -  093 .  174 

921  -  940 .  176 

979  -  1009 .  170 

1043  -  1077 .  180 

1  1  15  -  1  153 .  102 

1  195  -  1230 .  184 

1296  -  1335 .  186 

1309  -  1445 .  100 

1500  -  1573 .  190 

1645  -  1721 .  192 

1806  -  1096 .  194 

1998  -  2107 .  196 

2229  -  2363 .  198 

2515  -  2604 .  200 

2877  -  3095 .  202 

3349  -  3643 .  204 

3991  -  4407 .  206 

4916  -  5549 .  200 

6362  -  7439.  . .  210 

0941  -  1  1  173 .  212 

1404  0  -  %  22020 .  214 

42232  -  466033 .  216 


07-07-1986 


!  monthly  population: 

Popul at i on 

range  -  Sample 

Size 

291 

_ 

296 . 

125 

303 

- 

300 . 

127 

314 

- 

320 . 

129 

327 

- 

332 . 

131 

340 

- 

345 . 

133 

353 

- 

359 . 

135 

367 

- 

374 . 

137 

302 

- 

309 . 

139 

390 

- 

405 . 

141 

415 

- 

422 . 

143 

432 

- 

440 . 

145 

451 

- 

459 . 

147 

470 

- 

479 . 

149 

491 

- 

501 . 

151 

513 

- 

523 . 

153 

536 

- 

540 . 

155 

561 

- 

574 . 

157 

500 

- 

601 . 

159 

617 

- 

631 . 

161 

640 

- 

663 . 

163 

601 

- 

697 . 

165 

717 

- 

735 . 

167 

755 

- 

775 . 

169 

797 

- 

019 . 

171 

043 

- 

067 . 

173 

094 

- 

920 . 

175 

949 

- 

970 . 

177 

1010 

- 

1042 . 

179 

1070 

- 

1114 . 

101 

1154 

- 

1194 . 

103 

1239 

- 

1205 . 

105 

1336 

- 

1300 . 

107 

1446 

- 

1507 . 

189 

1574 

- 

1644 . 

191 

1722 

- 

1005 . 

193 

1097 

- 

1997 . 

195 

2100 

- 

2220 . 

197 

2364 

- 

2514 . 

199 

2605 

- 

2076 . 

201 

3096 

- 

3340 . 

203 

3644 

- 

3990 . 

205 

4400 

- 

4915 . 

207 

5550 

- 

6361 . 

209 

7440 

- 

8940 . 

21  1  ' 

1  1174 

- 

14847 . 

213 

22021 

- 

42231 . 

215 

465914 

and 

above  use  .... 

217 
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R5ED  V3.0 


Table  o-f  Adjustment  Factors 


FOR  DEFECT  RATE  ADJUSTMENT 


OVER  7.  -  THRU  7.  FACTOR  7. 

UP  TO'  -  1.0 .  0.46 

2.0  -  3.0 .  0.78 

4.0  -  5.0 .  1.00 

6.0  -  7.0 .  1.17 

8.0  -  9.0 .  1.31 

10.0  -  11.0 .  1.44 

12.0  -  13.0 .  1.54 

14.0  -  15.0 .  1.64 

16.0  -  17.0 .  1.72 

18.0  -  19.0 .  1.80 

20.0  -  21.0 .  1.B7 

22.0  -  '23.0 .  1.93 

24.0  -  25.0  1.99 

26.0  -  27.0  2.04 

28.0  -  29.0  2.08 

30.0  -  31.0 .  2.12 

32.0  -  33.0  2.16 

34.0  -  35.0  2.19 

36.0  -  37.0  2.22 

38.0  -  39.0  2.24 

40.0  -  41.0 .  2.26 

42.0  -  43.0  2.27 

44.0  -  45.0  2.28 

46.0  -  47.0  2.29 

48.0  -  49.0  2.29 


07-07-1986 

FDR  DEFECT  RATE  ADJUSTMENT 


OVER  7.  -  THRU'  7.  FACTOR  7. 

1.0  -  2.0 .  0.64 

3.0  -  4.0 .  0.90 

5.0  -  6.0 .  1.09 

7.0  -  8.0 .  1.24 

9.0  -  10.0 .  1.38 

11.0  -  12.0 .  1.49 

13.0  -  14.0 .  1.59 

15.0  -  16.0 .  1.68 

17.0  -  18.0 .  1.76 

19.0  -  20.0 .  1.84 

21.0  -  22.0 .  1.90 

23.0  -  24.0  1.96 

25.0  -  26.0  2.01 

27.0  -  28.0  2.06 

29.0  -  30.0  2.10 

31.0  -  32.0 .  2.14 

33.0  -  34.0  2.17 

35.0  -  36.0  2.20 

37.0  -  38.0  2.23 

39.0  -  40.0  2.25 

41.0  -  '  42.0 .  2.26 

43.0  -  44.0  2.28 

45.0  -  46.0  2.29 

47.0  -  48.0  2.29 

49.0  -  50.0  2.29 
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WHERE  SOME  DEFICIENCIES  ARE  NOT  REWORKED  WITHIN  ALLOTTED  TIME 


a.  Price  for  specified  services 

b.  Population  of  services  this  period  being  billed 

c.  Price  per  service 

d.  Number  of  services  sampled 

e.  Number  of  sampled  services  rejected 

f.  Observed  defect  (reject)  rate  [(e/d)xl00] 

g.  Adjustment  factor  Note  1 

h.  Deductible  defect  rate  (f  -  g)  Note  2 

i.  Extrapolated  defects  [Whole  number  portion  of  (h  x  b)/100] 

j.  Defects  observed  outside  the  sample  Note  3 

k.  Acceptable  rework  completed 

l.  Net  number  of  services  to  deduct  (i  -  k)  Note  2 

m.  Deduct  for  unsatisfactory  services  (c  x  1) 

n.  Liquidated  damages  (10*  of  c  x  k)  Note  4 

o.  Other  adjustments  (-  for  deduct)  Note  5 

p.  Total  Payment  (a  -  m  -  n  _♦  o) 

Notes-:  1.  Adjustment  factor  varies  with  observed  defect  rate. 


$S,000.00 
1,000 
$  5.00 

100 
5 

5.  00$ 
1.00% 
4.00% 
40 
23 
3 
37 

$  185.00 

$  1.50 
$  0.00 
$4,813.50 


2.  Cannot  be  less  than  zero  (0) 

3.  Defects  observed  outside  the  sample  will  not  be  used  as 
a  basis  for  extrapolation  but  shall  be  considered  in 
calculation  of  payment  for  satisfactory  rework. 

4.  Liquidated  damages  calculated  as  appropriate  in  accordance 
with  contract  provisions. 


5.  This  could  include  a  deduction  for  work  performed  by  Government 
or  other  forces  or  payment  for  rework  from  a  prior  billing  period 
provided  it  would  not  have  resulted  in  less  than  zero  Net 
Services  to  deduct  (item  1  above)  in  that  period. 
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